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ABSTRACT 

A project constructed a computer-based Test 
Development Center <COMBAT) . . It involved the development of 
instructional objectives and test items and their storage in a 
computer system from which they could be available to teachers. 
Project record data, questionnaires, and interviews were used to 
determine if the objectives were achieved, what the strengths and 
weaknesses of the system were, and how efficiently the project was 
run* Evaluation yielded the following results.. 1> A pool of 32,000 
items was developed, but greater depth and breadth were required. 2) 
Retrieval costs were too high« .3) The user communication system was 
effective. 4) In-service training and information disseminatioi) were 
inadequate. . 5) No evidence was obtained which showed a direct effect 
upon improving instruction. 6) Teachers needed to realize that COMBAT 
was only emergent, not completed. ,7) The management of the project 
was good. In summary, the project was moderately successful. . It 
showed a computer-based test development center could be built, but 
not that it effectively improved instruction. .Costs must be lowered, 
the item pool expanded, and teachers better informed about the 
system. (LB) 



FILMED FROM BEST AVAILABLE COPY 




COMPUTER BASED TEST 

Development 
Center 




COMBAT Evaluation Report 

Don Clayton, Project Director 



erJci 



us DEPARTMENTOF HEALTH, 
EDUCATION 

NATIONAL INSTITUTE C- 
EDUCATION 

THIS DOCUV.ENt MAS BCtN MEPRO 
OUCEO fcXACTLY AS srCHiVEC) J^»OV 
THE ffftSON 0« Of^GANlZATtON ORiG'N 
ATiNGiT POINTS or OPiNiO^iS 
STAtEO DO NOt NECESSAPUV ftEJ^ftE 
SENTO^nCiAL NATIONAL iNSTtTuTE Of 
EOuCAt»ON POStTiON OR ^Oi ifY 




Independent Ev«iluation Report 



COMPUTER BASED 
TEST DEVELOPMENT CENTER 



Don Clayton 
Project Director 



Report prepared by 

Robert Lange 
Frank Nelson 
Martin Birnbaum 



Teaching Research Division 
Oregon State System of Higher Education 



July, 1970 



TABLE OF CONTENTS 

Chapter Page 

I. THE PROJECT AND THE CONTEXT OF THE EVALUATION l' 

Overview of the Project 1 

Purpose of the Evaluation 4 

II. DATA COLLECTION AND ANALYSIS PROCEDURES 6 

Sources of Data • \ 6 

Sampling Procedure . 8 

Data Collection Procedure 10 

Data Analysis Procedure • • • a 12 

III. REPORT OF DATA RELATED TO PROJECT 

OBJECTIVES AND OPERATIONS 13 

Demographic Characteristics of the 

Teacher Questionnaire Respondents 13 

Objectives of the First TWo Years 20 

Objectives of the Third Year 70 

Perceived Project Potentials . 72 

Perceived Efficiency of Project Management 78 

IV. INTERPRETI\'E SUMMARY OF FINDINGS 90 

Congruency Between Objectives and Outcomes 90 

Contingency Relationships Within 

Project Operations and Outcomes . 92 

Special Problems Related to Project Context ..... 93 

V. GENERAL CONCLUSIONS 95 

APPENDICES . . 98 



Appendix A: System User Questionnaire 
Appendix B: . General Questionnaire 
Appendix C: MATPB Representative Questionnaire 
Appendix D: Cover Letters For Questionnaires 

1. System Users 

2. District MATPB Representatives 

3. District Superintendents^ COMBAT Service Area» 

Alert 

4. Elementary Principals, Non-user Questionnaire, 

Alert 

5. Secondary Principals, Non-user Questionnaire, 

Alert 

6. Delivery Instructions, All Principals, 
^ • Both Questionnaires 

?♦ Elementary and Secondary Non-user Questionnaire 

O 

ERIC 



LIST OF TABLES 

Table 

1. Demographic Characteristics of Respondents 

to Users* Questionnaire 

2. Grade Level of Teacher Users' I.D. Number 

3. Demographic Characteristics of Respondents 

CO General Questionnaire 

4. Multnomah County l.E.D, Classroom 

Teachers by Grade Level 

5. Teacher Users* Requests and Rating of Objectives 

6. Teacher User Requests and Rating of Objectives 

7* Sumnary of Time/Cost for Initial Requests 
_ ^ and Test Print -outs 

8. Teacher Rating of Final Test Prlnt-outs . . . 

9. Desired Elapsed Time- Between Initial 

Request and Receipt of Test 

10. Item/Test Selection and Receipt 

11. Tabulation of Teacher Requests . 

12 .^^^..^tendance at In-service Programs ...... 

13. 1 Teacher Testing Patterns 

14. Teacher Users* Testing Patterns 



iii 



LIST OF FIGURES 



Figure 



1. Cumulative Number of Items Included In CX)MBAT 
Files January 1, 1969 to June 1, 1970 • • , 



I. THE PROJECT AND THE CONTEXT 
OF THE EVALUATION 

A. OVE RVIEW OF THE PROJECT 

The original concept for the develoimient of a Computer-4}a8ed Test 
Development Center was developed by professional educators and 
researchers In the metropolitan area of Portland, Oregon, and Teaching 
Research — a Division of%the Oregon State System of Higher Education. 
This group received a planning grant from the Elementary and Secondary 
Education Title III project office In the fall of 1965. After consider- 
able planning and negotiation, this group obtained a Title III project 
grant In July of 1967 to develop a Computer-baaed Test Development 
Center (COMBAT). The project was designed to service schools in the 
Portland metropolitan area which consisted of the Portland Public School 
District and approximately 70 additional districts in MultnOTiah, -Clackamas, 
and Washington Counties — the three counties surrounding the city of 
Portland. The schools in this area have formed the Metropolitan Area 
Testing Program Board in an effort to Improve measurement of student 
characteristics and performance and curriculum evaluation procedures. 
On the basis of muttial agreen^nts between the agencies involved in the 
planning and the funding i^ency, the Multnomah County Intermediate 
School District was designated as the primary grant recipients The 
assistant superintendent for the Multnomah County I«E.D. was designated 
as project director. 

To facilitate the development and implementation of the project, 
the Multnomah County I.E.D. let two subcontracts. The first was for the 
development of the computer storage and retrieval system and for the 



operation and maintenance of the storage and retrieval system. The 
Portland Public Schools* computer center was the subcontractor for this 
actlvl^. The second subcontract was let to Teachli^ Research for 
development of the test Items and objectives to be stored in the computer 
system and for maintenance of communication between the agencies cooper- 
ating in the project. 

The overall purpose of the COMBAT project was to provide the test 
development and production service to classroom teachers* This service . 
was considered valuable because a typical testing program consisting of 
teacher-made tests » published or standardized tests, and locally developed 
tests, had deficiencies. These tests often lacked technical qtiallty, 
were of limited range in usefulness with small groups of students, or 
were not relevant to the content taught in the district. By using the 
large nusober of teachers as a source for test it^ns, the pro J act was to 
develop a pool of quality test items and objectivas that would be avail- 
able to individual classroom teachers for testing ^nd planning purposes. 
The project leaders also hoped to provide a quality service that would 
result in teachers using tests in a more diverse manner and for a wider 
variety of reasons, such as diagnostic testing, pretesting, and indivi- 
dualizing instructional programs. 

In order to achieve the overall purpose of the COMBAT project, 
large numbers of validated test items and Instructional objectives 
were to be stored in a computer. Teaching Research had the primary 
responsibility to assist teachers in writing objectives and test items 
for inclusion in the item pool. Project plana called for the selection 
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of a limited currlcular area and limited grade level for Initial Item 
and objective development. During later stages of the project the 
currlcular areas and the grade levels would be expanded. 

There were five general tasks for the project staff: 1) developing 
statements of objectives which encompassed the selected curriculum^ 

2) obtaining Items designed to measure particular objectives » 

3) storing test Items and objectives In a computer data bank» 4) organ*- 
izlng a system of communication between teachers and the C(MBAT facility » 
as well as dlssemlnatli^ relevant Information to the classroom teachers ^ 
and 5) producing^ on demand^ objectives and test Itemis in an acceptable 
form from teachers in a reasonably quick turn-around time. 

Classroom teachers were invited to participate in a variety of In- 
service and workshop programs in the production of test items and ob- 
jectives. These activities included evening and weekend workshops » 
stamner workshops » and extended workshops during the academic school 
year. Teachers were also Invited to contribute samples of the tests 
they used with their students as a basic source for test items and ob-^ 
j actives. The delivery system developed by the Computer Center con- 
sisted of a two-step9 tel^hone<*^ll procedure. Teachers made an initial 
contact with the COMBAT center by telephone and requested test items 
on specific currlcular areas for particular grade levels. A computer 
printout of the objectives and test items was mailed to the teachers. 
The second step required the teachers to contact the Computer Center by 
phone a second time to Identify specific items for a test. These items 
were then prepared by the computer into a test ditto which was mailed 
directly to the classroom teacher. 
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During the first half of the th..- 
. , y^"^ °' P^Ject. the major 

developmental effort was Halted to th. 

XMted to the Identification and production 
or test Items and oblectlveR -u 

Jectlves In the Social Studies for grade levels 
four, five, and six. Durins th* 

, , ^ Of the second year and 

during the third year of the project th. A i 

, , , project, the developmental effort expanded 

to include grade levels four through twelve m ' 

twelve, m lour currlcular areas: 

Social Studies, Lan8ua^e-Ar^B u u 

, i^nguage-Arts, Mathematics, and Science. 

B. mPOSE_OF THE EVALUATTnM 

primary function of this eva1..»^^ 
, , evaluation report is to fulfill the 

legal requirements for a summary evalu«r<o 

^ evaluation report as required of 

..e .... 

ESEA Tide II, a*,l„i„,.,i^ age„cl«. 

Ht:»cii.e, Che consortiun of pro- 

ject agencies; (c) developers of sirll.. 

. ^"^^^^^ Programa and other educational 

innovators. The administrative agencies of the Title III 

- U.S. Office Of Education and the State - o ""^^ 

roject. The contents of this evaluation should provide 
Information that will be useful to th«o 

useful to these agencies in completing their 

^^^^ 

- ...... 
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Computer Center. Although each of the cooperating agencies will use 
the Information In different ways, all of them will"* find aspects of the 
report that relate to the accountability criterion that are beglnnli^ 
to emerge In the educational research environment. In addition, an 
effort was made to identify some of the requirements for developmental 
efforts that are field centered or directly Involve classroom teachers. 
Outside observers, such as developers of similar projects and other 
educational Innovators, will find portions of this evaluation directly 
related to their planning efforts for field-centered developmental work. 

In order to serve these diverse purposes the evaluation was designed 
to answer three categories of questions. First, did the project attain 
its objectives as stated in the original and continuation proposals? 
Second, as perceived by teachers, school district testing representatives, 
and project personnel, what were the strengths, weaknesses, and greatest 
potentials of the COMBAT program? Third, as perceived by project repre- 
sentatives from each of the primary agencies associated with the COMBAT 
project, how effective and efficient were the project operational pro- 
cedures and project manag^ent? The data sources used in this effort 
include! (a) project record data, (b) field-tested questionnaires for 
the general teacher population served by the project, teacher users of 
the COMBAT system, and school district testing representatives, (c) tele- 
phone and personal interviews with key staff in the project consortium. 



11. DATA COLLECTION AND ANALYSIS PROCEDURES 

XT^ ^SOURCES OF DATA 

Rjicord Data: Project record data served two basic fimctions. 
First, portions were used in the direct evaluation of the attainment 
of projec** 'Objectives. Second, some records led to the construction 
of questionnaires and interview guides used with teachers and other key 
project personnel. The following list describes the major record data 
that contributed to the evaluation: 

1. Record of teacher users, by name and code number, indicating 
the number of first requests and the number of second requests 
made by each individual. A first request was defined to be an 
initial contact by a teacher for an individual list of objectives^ 
or test items. A second request was defined as a request for a 
test ditto of test Items selected from the Initial listir.g oi 
test items. Thus a teacher who made two distinct inquiries 
regarding separate topics but filled in eituer case to request 

a test ditto was recorded as having made two first requests 
and no second requests. At the same time, a teacher who made 
two separate inquiries on different content areas but requested 
a test ditto for only one of those areas was recorded as making 
two first reqiiests and one second request. 

2. A list of in-service programs conducted by the COllBAT staff. 
This data included a list of the names of the participants 
in each of the in-service programs and a statement of the 
objectives for each program. 
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3. The monthly reports, tabulated by the Computer Center, indicating 
the number of test items and objectives stored in the item bank, 
and the number and type of teacher requests made during that 
month. 

4. Copies of keyword indexes made available to teachers in the 
area serviced by the project. 

5. A list of the school districts included in the service area. 

6. A list of the agencies cooperating in the consortium and the 
key personnel associated with the project from each of those 
agencies. 

7. A description of the project functions designated for each of 
the participating agencies. This description was contained In 
the subcontracts let by the primary contracting agency, the 
Multnomah County Intermediate School District. 

8. Sampleu of objectives and test Items contained In the item bank. 

9. Copies of journal articles, presertations made at professional 
meetings, and other reports produced by the project staff. 

Questionnaires ; Three questionnaires were developed by the evalua- 
tion staff. The first questionnaire was daslgned for the general teacher 
population served by the COMBAT project. Items for this questionnaire 
were developed by the evaluation team on the basis of the record data 
and initial interviews with project staff. After the first form of 
this instrument was developed four members of the evaluation team 
visited four elementary and four secondary schools to field test the 
instrument with a sample of teachers In those buildings. The number 
of teachers included in this field test at each school varied from two 
through five. The instrximent was revised on the basis of the teacher 



response to the items and their comments regarding the format and con- 
tent of the Instrument, A copy of the final form of this general 
questionnaire Is Included In the Appendix, 

A second questlonnal-^e, COMBAT Users* Questionnaire, was developed 
and field tested In a manner slm:**!.: to *:hat of the general que :tlonnalrt. 
This instrument was used with teacuers who had used the COMBAT system. 
It differed from the general Instrument in length and in form. The major 
difference in form was the inclusion of five open--ended questions* A copy 
of this instrument is Included in the Appendix. 

The third questionnaire was designed for administration to the test** 
ing representatives from districts served by the project. This instrument 
was composed exclusively of open-*ended items. A sample of the question- 
naire is included in the Appendix. 

In addition to the three questionnaires , interview schedules were 
developed for use with key project personnel such as the members of the 
Teaching Research staff, members of the Advisory Board to the project 
as designated by the Metropolitan Area Testing Program Board, and the 
project personnel at the Portland School District Computer Center. 

B. SAMPLING PROCEDURES 

General Teacher Questionnaire ? Two general problems were associated 
with the distribution of the general teacher questionnaire. First, the 
project service area included a total of 266 elementary schools and 64 
secondary schools. Second, there was no complete list or frame of the 
teachers in each of these schools distributed in four regions, Multnomah 
County, Washington County, Clackamas County, and Portland. Since the 
project was directed toward grades four through twelve in four currl- 



culum areas 9 it was decided that the sampling procedures for the elemen- 
tary and secondary schools should be handled differently. 

The elementary schools in each of the three County Intermediate 
Education Districts and the Portland Public Schools were each assigned 
a number. The schools in the four areas were numbered separately. 
Through the use of a random number table, a 15 percent sample of schools 
was randomly selected independently in each of the four areas. That 
is, four independent 15 percent random samples of school buildings were 
selected. The school directories published by each of the four areas 
served as an indicator of the number of teachers in each building. 

In order to obtain an adequate representation from each of the 
four curriculum areas in the secondary schools a separate procedure was. 
needed. Each secondary school in the three Cotinty Districts and Port*- 
land Public Schools District was assigned a number. Fifteen percent of 
the schools in each of these four areas was randomly selected and assigned 
to each of four curriculum areas* That is, 15 percent of th^ schools 
in Portland Public Schools were assigned to English, 15 percent to math, 
15 percent to language-arts, and 15 percent to social studies. No school 
was assigned to more than one curriculum area. A similar technique was 
used for each of the County Intermediate School District regions. On 
the basis for the information published in the school directories, an 
approximation of the number of teachers in each department within those 
schools was identified. In this manner the sample included 15 percent 
of the teachers from each region in each of the four secondary curri- 
culum areas. It should be noted that in both the elementary and secon- 
dary school samples for the general questionnaire, individual teacher 
names could not be identified. Rather, the samples were taken on a 



school basis for the elmentary level and on a department basis for the 
secondary level, 

COMBAT Users' Questionnaire ; The Portland Computer Center supplied 
a list of the names and school buildings for every teacher that had made 
a formal request for test items or objectives. This list served as the 
sampling frame for the teacher users' questionnaire, A 25 percent 
random sample was selected from the list of users in each of the four 
regions; Portland, Multnomah Coxinty, Washington County, and Clackamas 
County • 

No sampling procedures were required for distribution of tl\e school 
district testing representives questionnaire nor the interviews v/ith the 
MAPTB COMBAT Advisory Board. Every testing representative was mailed a 
questionnaire and each member of the Advisory Board was interviewed. 

C. DATA COLLECTION PROCEDURES 

Record Data ; The record data were obtained through the Teaching 
Research COMBAT Project Coordinator. This procedure proved to be very 
satisfactory in most all cases. The major difficulty resulted in the 
type of record data sought by the evaluators and the availability or 
existence of specific types of records. 

General Questionnaire ; The distribution and return of the general 
questionnaire was hand'^.ed in a cooperative effort by the evaluators and 
the Intermediate Education District offices. An approximation of the 
number of teachers in each of the schools selected in the sample was 
obtained from the I.E.D. school directories. For each school, a packet 
was developed that included separate teacher envelopes containing an 
introductory letter and the questionnaire. These envelopes were packaged 
by school along with a letter of Instructions for distribution to the 
teachers. The packets intended for the schools in each of the County 



Intermediate Districts and Portland Public Schools were boxed separately. 
Each box was distributed to the appropriate I.E^D, office or to the 
Portland Public Schools Central Research Office. The responsibility for 
distribution of the individual school packets rested with the County Inter- 
mediate Districts and the Portland Public Schools Central Office. Once 
a packet was delivered to a school, the building principals had the 
responsibility to distribute the individual teacher envelopes to the 
appropriate personnel. The questionnaires were filled out and returned 
to the school principal who in turn packaged and returned them to the 
central collecting agency, that is, the County Intermediate Districts or 
the Portland Public Schools Research Office. In the case of the secondary 
schools the school packets were identified by department and the principal 
either distributed the individual teacher questionnaire packets himself or 
worked through the department chairman. 

Because an exact count^ of the number of teachers in each of the 
sample schools was not available, it is impossible to calculate an exact 
percentage of questionnaires that were returned. On the basis of the 
number of questionnaires sent out and the number of questionnaires re- 
turned, it is estimated that approximately 50 percent of the teachers 
responded. 

Teacher Users * (Questionnaire ; A teacher packet was prepared for each 
teacher in the sample. It contained a questionnaire and a letter that 
introduced the purpose of the questionnaire and the procedure for returning 
the document. Each envelope was addressed to a specific teacher by school 
building and district in which that teacher was working when he made the 
Initial request. The packets were collated by school building and then by 
district and county. The county packets were distributed to the County 
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lED offices and the Portland Public Schools Research Office. These agenclef^ 
distributed the packets to Individual schools and the principal was respon- 
sible for distributing the individual teacher envelopes to each member of 
the sample. After completing the questionnaires, the teachers returned 
the questionnaires to the school principals who in turn sent them on to 
the County I£D offices where they were collected by the evaluation staff. 
In a few cases teachers that were Included on the list of users were no 
longer teaching In the school building or district. Because these teacher:** 
could not be Identified^ it is difficult to calculate an exact return 
percentage. A total of 118 users* questionnaires were distributed and ^2 
were returned, a 35 percent return rate. It is estimated that the return 
percentage, after taking Into account teachers that were no longer in the. : 
respective school buildings, approached 50 percent. The procedures for 
making this estimate are described in the data analysis section. 

D. DATA ANALYSIS PROCEDURES 

The data analysis procedures for record data. Interviews, and queti- 
tionnalre responses are discussed In detail in the data analysis section 
In general these procedures Included a computer Item response tabulation 
for the questionnaires, and coding of interview responses from typed 
transcriptions. The procedures for analyzing record data varied dependin;.. 
upon the nature of the data. The Interviews obtained from the project 
personnel and from members of the Advisory Board to the COMBAT project 
from the Metropolitan Area Testing Program Board were either taped 
during the telephone interview or taped during personal Interview. 
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IIU REPORT OF DATA RELATED TO 
PROJECT OBJECTIVES AND OPERATIONS 



During the first two years the project activities were directed toward 
the attainment of six general objectives. These objectives centered on 
the development > design ^ and tmplementation of a field-centered ^ computer- 
based testing service. In the third year^ the project concentrated on 
four specific objectives focused on expanding in the development of the 
test item bank and increasing teacher use of uhe system. 

In order to make the report for the evaluation data maximtjmly useful 
to the reader^ the data analysis is divided into five sections. The first 
section describes the demographic characteristics of the respondents tc 
the teacher questionnaires. The remaining four sections report the data 
that are directly related to the objectives of the first two years ^ the 
objectives of the third year » project management, and perceived project 
potentials. 

A. DEMOGRAPHIC CHARACTERISTICS OF THE 
TEACHER QUESTIONNAIRE RESPONDENTS 

The degree to which the respondents represent the total population 

is a critical concern in the interpretations of questionnaire data 

obtained through the mails or similar delivery system. It is doubly ^ 

difficult to address this issue in the current report since no adequate 

description of the general teacher population or the teacher user 

population was available. To help establish the credibility of the 

data reported herein, it was deemed necessary to collect and report a 

limited amount of demographic data from each respondent. Tables 1 and 3, 

respectively, contain a summary of teacher responses to six demographic 
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questions on the teacher users* questionnaire and the general teacher 
questionnaire. 

Two Items of record data were available as referents for the data 
presented In Tables 1 and 1. The first Is a tabulation of the grade 
levels Included In the code number assigned to each teacher user. This 
grade level corresponds to the grade level of the test items requested 
by each teacher user on his initial contact with the COMBAT system. 
A tabulation is presented in Table 2« The second referent is a tab* 
ulatlon of the number of teachers assigned to each grade level in the 
school districts serviced by the Multnomah County Intermediate Educa-- 
tlon Dii^trlct during the school year 1969-70 (see Table 4)« 

The data in Table 2 can be compared with the data contained in row 
five of Table 1. That is, the data headed by "grade level of the 
majority of your students." It can be seen from Table 1 that 72 percenf 
of the teachers responding to the users* ques^tionnaire taught ir» grad?: 
levels one through six^ whereas data in Table 2 indicate that 64 percent 
of the users' had ID numbers associated with grade levels one through 
six. In a similar manner, it can be noted that 28 percent of the ques- 
tionnaire respondents indicated they taught in grade levels nine through 
twelve, whereas 36 percent of the teacher ID numbers were associated 
with grade levels nine through twelve. Although this evidence is 
tenuous, it does indicate that respondents to the teacher users* ques- 
tionnaire nearly approximates the grade level percentage breakdown as 
recorded on the teacher users* ID numbers 

Four of the items contained in Table 1 need further examination* 
First, it should be noted that the breakdown of the sex variable as 
reported in Table 1 does not represent a consistent proportion by grade 
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TABLE 1 



Demographic Characteristics of 
Respondents to Users' Questionnaire 



Sex 



36% 
15 
Male 



64% 
11 
Female 



Age 


24% 


29% 


19% 


22% 


5% 






10 


12 


8 


9 


2 


1 




20-30 


31-40 


41-50 


51-60 


61-over 


N.R. 


College Training 


19% 


24% 


22% 


19% 


13% 




(degree & term hrs.) 


8 


10 


9 


8 


7 


0 




B.S. 


BS+20 


MS or 


MS+20 


^B+45 


N.R. 








BS+45 








Major Teaching 


57% 


2% 


2% 


8% 


10% 


24% 


Assignment 


24 


1 


1 


3 


4 


10 




Self- 


Science 


Soc. 


Math 


Lang. 


Other 


Contained 




Studies 




Arts 




Grade level of 


12% 


60% 


10% 


2% 


5% 


12% 


majority of students 


5 


25 


4 


1 


2 


5 


you teach 


1-3 


4-6 


7-9 


9-10 


9-12 


11-12 


Grade levels 


67% 


8% 


5% 


5% 


14% 




included in 


28 


3 


2 


2 


6 


1 


your school 


1-6 


6-8 


7-9 


9-12 


10-12 


N.R. 



N.R. 



N.R. 



TABLE 2 

Gr^de Level of Teacher Users' I.D. Number 



Grade level 



4% 
20 
1-3 



60% 
282 
4-6 



24% 
112 
7-9 



12% 
56 
10-12 
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level. In grades one through six the percentages were approximately 
30 percent male and 70 percent female. Whereas in grades seven through 
twelve the percentages were approximately 60 percent male and AO percent 
female* Second, the age distribution of teachers in grade levels one 
through six was approximately equally divided over the ten year age spar^ 
£rc«n 20 through 60; that is, approximately 20 percent in each of the fou" 
ten year spans. On the other hand, the teachers In grade levels seven 
through twelve concentrated in the age span 31 through 40 with 50 percent 
of these teachers In that age category. Third, approximately 50 percent, 
of the teachers in grade levels one through six had received no more than 
their Bachelor ^s plus 20 hours of college training. In grade levels seven 
through nine, 75 percent of the teachers reported they had received at 
least a Master's degree. Lastly, it should be noted that all of the 
teachers who Indicated they taught in self-*contained classrooms were 
teachers in grade levels one through six Five ot the ten t<^.^.:hrrs wb : 
Indicated their majcr teaching assignment as ''other,^* taught at grade 
levels one through six. 

Table 4 describes the distribution of teachers by grade level in 
school districts served by the Multnomah lED during the 1969*70 school 
year. Under the assumption that the school districts serviced by the 
COMBAT project have a distribution of teachers by grade level that Is 
approximately similar to that represented in Table 4, we can compare 
Table 4 with the fifth row of Table 3. In making this comparison two 
points should be. kept in mind. First, the directions for the distribution 
of the general questionnaire, initructed the principals to distribute th'^ 
questionnaire only to teachers in grade levels four through six, and to 



teachers in the four academic areas for which items were included in the 
item bank. Secondly, the information contained in Table 4 includes all 
secondary teachers; that is, those assigned to the four academic areas 
corresponding with the project activities as well as those in other 
curticular areas. On the basis of these last two points, it could be 
expected that the percentages of teachers in the lower grade levels 
responding to the general questionnaire would be slightly higher than 
the percentages recorded in Table 4. Likewise, the percentages of 
teachers in the secondary grade levels responding to the general ques- 
tionnaire would be slightly lower than the percentages recorded in Table 
4. By examining Table 4 and the fifth row of Table 3, it can be seen 
that these expectations were fulfilled. Although this Information is 
very sketchy at best, it does help to indicate that there was no gross 
systematic bias in the return of the general questionnaire on the basis 
of grade level. This last statement is based on the fact that 43 percent 
of the respondents to the questionnaires taught at grade levels one 
through six, whereas 31 percent of the teachers in grades four through 
twelve in Multnomah County taught in grades four through six. Likewise, 
25 percent of the respondents indicated they taught in grades seven 
through nine, wheread a total of 25 percent of the teachers in the 
Multnomah County area taught in either grades seven through eight or 
the junior high level (after excluding grades one through three). 
Likewise, 32 percent of the teachers responding to the questionnaire 
indicated they taught in grades nine through ten, nine through twelve, 
or eleven through twelve, whereas 44 percent of the teachers in 
Multnomah County from grades four through twelve were identified as 
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teaching in the high school level. A closer examination of the data 
reported in the first four rows of Table 3 showed trends by grade level 
that were very similar to those reported for the teacher users' 
questionnaire* 

There are some obvious discrepancies that appear when the data 
presented in Tables 1 and 3 are compared. These discrepancies are best 
explained by reminding the reader of the operation of the project. 
During the first two years the project effort was centered on obtaining 
objectives and test items for grades levels four through six. Only in 
the later stages of the project, that is the third year, was a concen- 
trated effort made to expand the item pool to include items for grade 
levels seven through twelve. It should also be noted that the majority 
of the effort of the project during the first years concentrated on 
social studies in the lower grade levels » and that Science, Mathematics, 
arid Lang jsge Arts subject area 3pecisli*5ta in the higher grada J ovals 
were not encouraged to use the system until the third year of the projec 
With these con'iitions in mind the discrepancy ii the sex variable betwa; 
the two tables can be explained on the basis of the concentration of 
female teachers in grade levels four through six. Thus, the teacher 
users' data indicates a higher percentage of females than indicated in 
the data from the general teacher's questionnaire. It is not feasible f 
attempt to analyze any discrepancy between the two sets of data on the 
college training of the teachers. There was a high percentage of non-- 
respondents to this item on the general questionnaire. This high percent 
of nonresponse was due to a typographical error in the printing of the 
questionnaire. Since the general questionnaire was sent to a sample of 
teachers in all areas serviced by the project at the close of year three, 



TABLE 3 



Demographic Characteristics o£ 
Respondents to General Questionnaire 



Sex 



48% 51% 

224 243 5_ 

Male Female N.R. 



Age 



33% 
156 



23% 
109 



22% 
106 



15% 
69 



6% 
29 



20-30 31-40 41-50 51-60 61-over N.R. 



College Training 
(degrees & term hrs.) 



8% 
41 



B.S, 



28% 
128 
BS+20 



34% 
161 
MS or 
BS+45 



5% 
25 



MS+20 



5% 
27% 



MS+45 



19% 
90% 



N.R. 



Major Teaching 
Assignment 



44% 
209 



5% 
26 



Self- Science 
Contained 



21% 
99 



12% 
56 



Soc. Math 
Studies 



12% 
53 



Lang. 
Arts 



5% 
26 



Other 



N.R. 



Grade level of 
majority of students 
you teach 



4% 
21 



1-3 



39% 
185 
4-6 



25% 
118 
7-9 



12% 
54 



9-10 



13% 
59 



7% 
33 



9-12 11-12 



N.R. 



Grade levels 
included in 
your school 



51% 
245 
1-6 



7% 
35 



6-8 



11% 
49 



7-9 



5% 
24 



9-12 



23% 
112 
10-12 



N.R. 



TABLE 4 

Multnomah County I.E.D. 
Classroom Teachers by Grade Level 
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26% 23% 11% 7% 33% 

356 323 151% 102 456 
Grade Level 1-3 4-6 7-8 Jr. Hi. Hi. Sch. 



% by Grade Level 31% 15% 10% 44% 

after excluding 4-6 7-8 Jr. Hi. Hi. Sch. 



grades 1-3 
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it was expected that the percentage of respotises for each category on 
the remaining items reported in Tables 2 and 3 would not be in close 
agreement. Again this expectation vas due to the fact that the two 
populations were similar only during the third year of the project. 
The responses to the users' questionnaire for these items were heavily 
weighted by teachers who used the system during the first two years when 
the target population was restricted. 

To further check on the credibility of the data reported herein, it 
should be noted that record data Indicated only one~third of the teacher.^ 
making a request for objectives and test items ever requested a test 
ditto. In response to an It&a on the Teacher Users* Questionnaire, 33 
percent of the respondents reported they had ordered a test ditto. 

B. OBJECTIVES FOR THE FIRST lUO Y^S 

Objective 1 ; To Identify the specific objectives of instructi on '1 . 
all elementary and secondary schools curricular areas for sll grade leve j 
This first goal of the project is not clearly defined. Given the large 
number of school buildings Involved in a three-county area plus the 
Portland Public School District, one can expect a tremendous variability 
in the objectives for similar courses at a given grade level. An attempt 
to identify specific objectives of instruction in all curricular areas 
for all grade levels is an extremely large task. In addition, there 
are no criteria that exist to determine the extent to which all of the 
objectives have been identified. From an operational viewpoint, during 
the major part of its first two vears, the project limited its effort tc 
identifying specific objectives in social studies for grade levels four 
through six. During the later portions of the second year and during 



ERIC 



- 20 - 



the third year the attempt to identify specific objectives was expanded 
to include the curricular areas of Mathematics, Social Studies, Language 
Arts, and Science for grade levels four through twelve • 

The project staff identified practicing teachers as the chief 
source for obtaining behavioral objectives. In order to collect these 
objectives, teachers were invited to participate in work sessions, for 
writing objectives and test items, during the winter and early spring of 
1967-68. During the siammer of 1968 the project staff conducted two 
teacher workshops in the construction of behavioral objectives and writing 
of test items. During the winter of 1968-69 the staff worked with classroom 
teachers through a Department of Continuing Education credit class entitled 
Measurement of Educational Objectives. During these classes teachers were 
actually involved in writing objectives and test items. The initial effort 
to obtain objectives and test items proved to produce needed quantities of 
objectives and items at far too slow a rate* By April of 1968 the work 
session approach had provided only a total of 500 objectives. The stimmer 
workshop session was much more productive. Unfortunately, an exact count 
of the number of objectives written during these sessions was not main- 
tained. The primary source of evidence of the productivity of the Stimmer 
sessions and ^^f the credit courses offered during the Fall and Winter 
of the 1968-69 year consist of a record beginning in December of 1968. 
This record listed the total nimiber of items and objectives contained in 
the data files. By December of 1968 the number of items had jimiped to 
2,830 and by April 1969, the number had reached 6,447. These latter 
figures do not indicate the exact number of objectives that were con- 
tained in the system because each test item contained in the record file 
was also associated with an objective. In some cases a single objective 



was associated with more than one test Item. Near the end of the second 
year of the project It became obvious to the staff that other procedures 
would have to be used In order to expand the number of objectives and 
test Items in the system. One of the primary procedures used was to 
request teachers to contribute sample copies of tests they used lu thei; 
Instructional programs. The staff used these teet items as a basis to 
infer the objectives. An instructional objective was written by the 
developmental staff for each test item selected from this item pool for 
inclusion in the item bank. This last procedure markedly Increased the 
rate at which objectives were identified. By September of 1969 the 
number of test items and associated objectives Included in the data files 
increased to 10,191. Through December, 1969, the number of items had 
jumped to 24,180 and by June, 1970, the data files contained 32,163 item > 
Again it should be remembered that these numbers do not indicate the 
n^jaiber separate objectives but' the ntmber of separate tnst Iters. 
There is no available data to indicate the exact number of separate 
objectives contained in the data bank. 

Overall, the identification of specific instructional objectives 
has been a major problem with the oroject. The developmental staff 
found it very difficult to find teachers who were trained to think about 
ard utilize instructional objectives at the level of specificity that 
had been selected as desirable for the data bank. Teachers were asked 
to write objectives in a form that specified the au^llence or students 
for whom the objectives were most appropriate, a specific behavior, the 
conditions under which the behavLor was to occur, and the degree or 
level of acceptable performance. The decision to write the objectives 
at that level of specificity resulted in objectives that differed very 



little from the test items themselves. The two members of the project 
staff sharing the primary responsibility for directing the developmental 
effort identified the form of the objectives as a major concern* One 
director identified the decision to store the objectives in specific 
format as "...most critical since it would be difficult to change the 
system after several thousand items were included. They ended up being 
very specific and, thus, not beii^ usable by most classroom teachers 
right now. The objectives are almost test items." When asked to discuss 
the decision to write behavioral objectives in specific form, the second 
director stated, "It became obvious that it would never work. Teachers 
don't have time to generate behavioral objectives, can't identify objec- 
tives, but given lists they may be able to select objectives and generate 
curricula." lie went on to state, "I bet that teachers don't pay much 
attention to the objectives because they are only slightly different 
from the test items." The problem associated with getting teachers to 
identify behavioral objectives was further varified through interviews 
with the COMBAT Advisory Board. When asked the question, "Do you 
think that both objectives and test items are equally important for 
teachers?" Three of seven Board members indicated that from a practical 
point of view, at the present time test items are more important. They 
went on to say they felt that teachers would eventually recognize the 
importance and utility of the objectives. When asked what they thought 
some of the ways chat the system would manifest itself in the classroom, 
two of the Board members indicated that teachers would not begin to use 
the objectives until after they had more experience with the use of the 
test items. 
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The record and interview data described above indicate the project 
was able to identify a large number of instructional objectives. However » 
there were major problems in getting teachers to write objectives and 
to use the objectives in their instructional programs. As a result, the 
project staff were not able to adequately identify objectives for all 
grade levels in all four curriculum areas included in the project. 

Further evidence on the attainment of goal was obtained through 
responses to the teacher users' questionnaire. This questionnaire 
contained two structured response items related to the objectives con- 
tained in the COMBAT file. The responses to these items are reported 
in Table 5. Examination of this table indicates that few teachers 
requested objectives more than once or twice. Interviews with the star' 
operating the Center for receiving the teacher requests indicated that 
in most all cases teachers requested test items at the same time that 
they requested objecti^/es. Very seldom -itt teachers request O'llv ob- 
jectives, however, the records of this information were not maintained * 
Table 5 also indicates that the majority of the teacher users who 
responded to the request to rate the appropriateness of the objectives 
rated them as approximately 50 percent appropriate. Of the remaining 
teachers who responded to this item, none rated the appropriateness of 
the objectives as very high and approximately an equal nximber of teacherr^ 
rated them as either high or low and very low. When asked to indicate 
their reasons for requesting objectives, nearly 60 percent of the user 
respondents indicated their reason was to experiment with the COMBAT 
system. Slightly less than 20 p ircetit of the respondents indicated that 
they ordered ob actives to use as a guide for ordering test items. 
Likewise, only 10 percent of the respondents indicated that they requested 



TABLE 5 



Teacher Users' Requests and 
Rating of Objectives 



How many times 17% 57% 14% 5% 0% 

have you requested 7 24 6 2 0 3 

a list of objectives? 0 1-2 3-4 5-6 7 or N.R. 



more 



Rate the appropriate- 8% 10% 43% 14% 0% 26% 

ness of the objectives 3 4 18 6 0 11 

for your intended very low 50% high very N.R. 

purpose, low appropriate high 



Reason 



No. 



Which of the following 
best matches your reason (s) 
for requesting objectives? 



(a) To experiment with 24 57% 
COMBAT. 

(b) To use objectives 4 10% 
as a guide for 

planning instruction. 

(c) To search for new 11 26% 
ideas for classroom 
instruction. 

(d) To use objectives 7 17% 
as a guide for 

ordering test items. 



(e) Other 



8% 



objectives as a guide for planning instruction. Since two of the primary 
reasons for including objectives in the item pool were to help teachers 
plan instruction and to serve as a guide for ordering test items, this 
data indicates that teachers seldom use the objectives for this purpose* 
However, nearly a quarter of the teachers found the objectives useful 
for searching for new ideas to be used in their classroom instruction. 

When teacher users were asked to make "one positive comment and 
one negative comment about the procedures for obtaining objectives, 
list of test items and test masters," many of the comments related to 
the quantity and quality of the items, rather than to procedures. Two 
elementary school teachers stated that the objective and test items 
were "excellent" and "well written." One teacher stated Chat there 
were "many items to choose from" and another said "Helps get other 
teachers testing questions (not narrowed)." Of the nine negative 
coCTsr.tr. ^lAde by secondary school teachers^ ^ix centered on tb^ l^t^^^ 
of sufficient breadth and depth of the objectives and items stored In 
the item bank* Of the twenty-eight negative comments made by elementar^, 
school teachers, five teachers indicated that the objectives were "not 
relevant" to these instructional programs. Another five teachers 
commented on the lack of sufficient items. Two other teachers commented 
on the quality of the test items by indicating the items were "too 
specific." The remaining comments to this question were directed at 
the procedures for using the system. 

Objective 2 ; To develop a v>ool of test items appropriate for asses3.,n ^4 
the attainment of the instructional objectives as identified . Because of 
the close association between the identification of specific objectives 
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and the writing of test items much of the data reported above is appro- 
priate for the second objective as well as the first. Tlie record data 
on the number of test items and objectives is a more accurate accounting 
of the number of separate test items than the number of objectives. As 
previously mentioned, there were a total nf 32,163 items contained in 
the COMBAT file as of June, 1970. Figure 1 illustrates the month by 
month rate of development of the item pool. Since this monthly tab- 
ulation was not initiated until the first of December, 1968, Figure 1 
fails to illustrate the increased rate of production of test items 
resulting from the Summer workshops of 1968 and the classes in Measure- 
ment of Behavioral Objectives taught in the Fall and Winter 1968-69, 
as compared to the rate of production of test items during the Winter 
and Spring of 1967-68 academic year. The marked increase in the number 
of items stored in the COMBAT files between the months of December, 1969 
and J^?*Turiry, 1970 was duo basically to the coer-'ations at the conputtiv 
center. In the fall of 1969 items were being written at a rate slightiv 
f aster than they could be key punched and stored in the file. During c . 
month of December the Computer Center hired temporary staff to catch up 
on the backlog of items that were written and waiting to be key punched 
and stored in the file. 

The production of a number of test items is only one facet of the 
second objective of the project. The second facet relates to the 
appropriateness of the items for assessing the attainment of instructional 
objectives. The teacher tisers' questionnaire contained four structured 
response items relating to this facet of the objective. The tabulation 
of the responses to these items is reported in Table 6. As indicated 
by the first item in Table 6, slightly over a third of the teacher 
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respondents failed to rate the appropriateness of the test items for 
the associated objectives. Of those that responded, approximately half 
rated the items as 50 percent appropriate and eight teachers rated the 
appropriateness as high or very high, whereas only four rated them as 
low or very low. When asked to rate the test items received from the 
COMBAT center on item difficulty, range of content, range of item style, 
and clarity, most of the teacher respondents checked the rating scale. 
As indicated by the responses tabled to the second question in Table 6, 
approximately half of the teachers indicated that the item difficulty, 
the range of content, and range of item style were about right. Approxi- 
mately a quarter of the remaining teacher respondents rated the item dif- 
ficulty as hard, the range of content as similar, and the ranges of item 
style as similar. Approximately three quarters of the teachers rated the 
items as clear. Six teachers indicated that the items were ambiguous, while 
two rated the items as very clear. The third question in Table 6 indicates 
that slightly over 60 percent of the teachers felt that only zero to 20 per- 
cent of their total classroom testing could be done with test items obtained 
from the COMBAT system. An additional 30 percent of the teachers indicated 
that 20 to 60 percent of their total classroom testing could be done with 
test items obtained from the COMBAT system. These percentage breakdowns 
were approximately the same for teachers at the elementary level and second- 
ary level. The tabulation for the fourth question in Table 6 indicates 
that approximately one-third of the teachers selected items from the list 
provided by the COMBAT center and requested a test ditto. At the other 
extreme, approximately a quarter of the teachers who requested test items 
from the COMBAT center reported that they did not use the items with students. 
The majority of the remaining teachers Indicated that they use test items 



TABLE 6 



Teacher User Requests and Rating of Objectives 



If you requested 
both objectives and 
associated test 
items, RATE the 
/J^PROPRTATENESS of 
the test items for 
fhe oH-tectives* 



3% 



very 
low 



8% 



29% 
12 



14% 



low 50% high 
appropriate 



5% 

2 



very 
high 



3% 
1 



did not 
rf-ques t 
both 



Rnte ih? test iters 
you received on each of 
the following criteria: 



ambiquous guous 



Approximately what percent 
of your total classroom 
testing could be done with 
test items obtained from 
the COMBAT system? 



62% 
26 



10% 
4 



clear 



19% 
8 



5% 
2 





difficulty 


3% 


27% 


55% 


10% 


0% 


8% 






1 


11 


23 


4 


0 


3 






very 


hard 


about 


easy 


very 


N.R. 






hard 




right 




easy 




b) 


range of content 


5% 


29% 


38% 


12% 


8% 


10% 


2 


12 


16 


5 


3 


4 






very 


similar 


about 


diverse 


very 


N.R. 






similar 




right 




diverse 




c) 


range of item 


5% 


27% 


50% 


10% 


3% 


8% 




style 


2 


11 


21 


4 


1 


3 




very 


similar 


about 


diverse 


very 


N.R. 






simi lar 




right 




diverse 




d) 


clarity 


0% 


14% 


74% 


5% 


8% 










6 


31 


2 


3 








very 


ambi- 


clear 


very 


N.R. 





0% 
0 



0-20% 21-40% 41-60% 61-80% 81-100% 



What did you do with the list of 
test items received from the 

COMBAT center? No> %_ 

a) Did not use with students. 11 27% 

b) Selected a few items and added a few of your own 4 10% 

before preparing a test ditto. 

c) Selected several items and added several of your 5 12% 

own before preparing a test ditto. 

d) Selected items only from the list and prepared 6 14% 

your own ditto. 

e) Selected items only from the list and ordered 14 33% 

a test ditto. 

r > Response 2 5% 



from the COMBAT center and either prepared their own ditto directly from 
this list or added some items of their own and prepared their own test 
ditto. Overall this ip^ans that approximately 70 percent of the teachers 
who requested test items frcrai the COMBAT system found them to be useful 
in their testing programs. Since the item pool was in a continuous state 
of development through the total project, this result is a strong indica- 
tion that the project was becoming functional for the teacher users. 

Overall, the data indicate that the developmental staff were able 
to collect a large number of items for the item file. Thirty-two thousand 
items represents a great deal of production. However, there is clear 
evidence that this item pool is far too small to make the project maxi- 
mumly functional in four curriculum areas grades four through twelve. 
The procedures used to produce items was sufficient to obtain items that 
tended to be rated approximately 50 percent appropriate by the teachers. 
Two problems probably prevented teachers from rating the appropriateness 
of the items as high or very high. The first is the wide diversity of 
instructional approaches used by different teachers at the same grade 
level and in the same content. The second dealt with the mechanics of 
selecting items from the item pool. This second problem will be dis- 
cussed in more detail under the third objective. 

Objective 3 ; To design and operate a computer center necessary for 
the implementation of the test development activities . This objective 
has two basic components. The first concerns the development of a computer 
system to handle storage and retrieval of instructional objectives and 
test items. Included in the evaluation of this component is an examina- 
tion of the interaction between the computer staff and the developmental 
staff, as well as an investigation of the computer time required to produce 
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the list of test items and tests requested by the teachers* A second 
component concerns the adequacy of the telephone-mail communication 
system between teachers and the computer center, and a determination of 
the teacher ratings of the print-out format for objectives and test items. 

The primary source for information on the development of the compute- 
system was a set of interviews with the staff at the computer center and 
the developmental staff. Unfortunately ^ there was no systematic documen- 
tation of the procedures and problems encountered during the developmental 
activities to serve as a basis for evaluative information. All but one 
of ♦•he school district testing representatives to the MATPB and all but 
one of the members of the COMBAT Advisory Board declined to comment on 
the operation of the computer system on the basis of a lack of knowledge 
of computer problems and operation « 

The Portland Public Schools* Computer Center subcontracted with the 
Muicncmah County Intermediate Education District to construct and oper-^^ 
the computer system* The Computer Center built the COMBAT system to 
complement existing computer systems that serviced a number of other 
projects in the Portland School District* This decision allowed the 
Computer Center to take advantage of a good deal of development work that 
had been done in relation to other projects. The original system was 
written in machine-level assembly language coding for the Honeywell 200 
Series Computer. Instructional objectives and test items were stored on 
tapes in natural language format. A classification scheme was built so 
that each test item or objective could be coded by grade level and by a 
selected set of key words. This permitted the items to be retrieved by 
grade level and on the basis of key words. In addition^ the natural 
language storage permitted the development of the capacity for retrieving 



an objective or test item by searching the items for specific words. 
Therefore, when a teacher makes a request for items on a specific content 
that was not included in the set of key words it is possible to search 
each item stored in the file for specific words that were not included 
in the master list. 

The Computer Center staff reported the chief advantage of the 
system as its ability to store and retrieve items before the development 
of a comprehensive code. Other advantages of the system were its flexi- 
bility in storage and editing capacities* The system permitted a wide 
assortment of printout formats available for use on computer printouts 
sent to teachers. However, the center staff reported they actually used 
only a small number of variations in the printout format. In practice 
the number of formats was limited to two or three. Another unique feature 
of the system is its capability to identify auxiliary materia]-, such as 
maps and charts that were to be used with specific items. These maps and 
charts were coded and could be retrieved very easily for distribution to 
teachers along with the computer print-out material. 

Limitations of the system, reported by the computer staff, are first, 
the system is not easily transferable to other machines* Although such a 
transfer is not impossible. It would require a significant amount of work. 
The use of the natural language storage was less efficient than the alter- 
native of storage in a coded form. That Is, it takes a greater amount of 
time to retrieve the items stored in the natural language than If the 
items were stored in a coded form. Altjiough the system could very easily 
handle constructed response items and objective test items such as multiple 
choice and true-false, there were some problems with storage of matching 
Items and limitations on the type of Items that could be stored in the 
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system. A particular problem existed with storage of items relating to 
mathematics and technical fields such as chemistry and physics where 
special symbols with superscripts and subscripts were used. This prob- 
len existed because the printer did not have all of the characters needed 
to print mathematical or equation-type questions. In addition, only one 
size of character was available. Therefore, some equation test items 
Were difficult to store and retrieve in the form desired by the teachers. 
Although one advantage of the system was Its capacity to search individual 
items stored in the file for a specific word, this capability also pro- 
duced some problems. Examples of problens that resulted from the search 
of individual items are as follows; occasionally the words being searched 
were contained In the dlstractor portions of multiple choice items and 
misspellings or nonstandard word usage prevented some items from being 
retrieved. As a result of these problems. Items that were inappropriate 
for a particular request were sometimes retrieved. A second type of pre 
lem which exists is the lack of teacher specificity In topic or content 
for which they wished items retrieved. Teachers were allowed to select 
the maximum number of items they were to be sent. If the system ldentifit»d 
200 items and the teacher requested 50 items to be sent for examination, 
the computer would randomly select from the 200 items identified. The 
random selection cf 50 items were printed and mailed to the teachers. 
This procedure sometimes prevented teachers from getting as wide an assort- 
ment of Items as they may have wanted and, because of the reasons stated 
above, occasionally selected items that were inappropriate for the requests 

The system retrieval operations were set up so a teacher could make 
a request by phoning into the Computer Center and stating the grade level 
anc! the topic for which she desired Instructional objectives or test Items. 
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Although an advantage of this procedure was the ease with which the 
teachers could obtain sample items or objectives, it presented some prob- 
lems. One of the major problems existed in the lack of specificity with 
which teachers were able to describe their request. The operation of 
this system was facilitated greatly by the telephone request receivers 
in the Computer Center. These persons had a thorough knowledge of the. 
system and were able to help teachers specify their content in the manner 
that was most compatible with the items stored in the system. A limitation 
of this procedure is that it requires a highly competent person to receive 
the request. This -^erson must know the system and its contents as well as 
have the ability to interact with teachers in a way that assists the 
teachers In specifying the exact nature of the material for which t.aoy were 
testing. 

During operation of the project an alternative request procedure was 
tried. This procedure required teachers to mail a card that contained a 
description of the content for which they wanted test items. The mailing 
procedure proved to present many problems, with the central problem being 
the lack of specificity in teachers' requests. For example, a teacher 
might put on a card simply that they wanted test items in the area of 
biology. Such a request was much too general and would generate a tre- 
mendous number of items. Frenuently only a small number of these items 
actually matched the teacher's need for test items at that time. Frequently 
the teachers who were making requests by mail had to be contrcted by the 
phone in order to delimit the nature of the request. 

ihe '^omputer Center was only one of several key educational organi- 
zations involved in the project. The primary responsibility for maintaining 
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communication and other interactions between the different educational 
organizations was with the developmental staff; that is, the staff at 
Teaching Research. The task of interfacing the activities of the devel- 
opmental staff and the computer staff centered on adequate communication 
between the two agencies. Although the staff in both agencies indicated 
there were no major problems in communication, they did identify a few 
smaller problems. First, there was no direct contact between the Com- 
puter Center and the Metropolitan Area Testing Program Board. None of 
the members of the Computer Center staff indicated that they knew of the 
operation or intents of the MATPB program. There were a few probleirc in 
obtaining feedback information from classroom teachers on the content and 
quality of the items stored in the item file. Although procedures for 
obtaining feedback from the teachers were set up by the developmental 
staff, few teachers took the opportunity to provide information via this 
route. In addition, only a small percent of teachers contacted the Com- 
puter Cenzer after their first request. Part of the procedure of the 
second contact was to identify inappropriate items and to identify errois 
such as misspelling and incorrect or improper wording and structure of 
test items. Although these procedures did provide some feedback for the 
refinement of the items stored in the system, this feedback was smaller 
than that desired by the Computer Center staff. 

Because the item files were stored in the natural language form, 
a large amount of computer time was required in order to produce an 
individual list of tes^ objectives or test items for a teacher request. 
The actual amount of time required to produce a single list of objectives 
or items varied with the number of requests. Depending upon the complexity 
of the request, a single request could take anywhere from 20 to 30 minutes 
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of computer time. At the same time it should be noted that the system 
was built so that requests could be processed in batches. The amount of 
time per request was markedly reduced when the requests were batch-proc- 
essed* Two similar requests could be batch-processed in less than twice 
the time of a single request. This means that two or three requests did 
not take two or three times as much computer time ns a single request* 
For example, the records at the Computer Center indicated that where a 
single test may require from 20 to 30 minutes of computer processing 
time seven tests on one occasion were processed in 1 hour and 45 minutes. 
On separate occasions four tests were processed in 2 hours and 20 
minutes, eleven tests in 3 hours and 51 minutes, and seven tests in 1 
hour and 45 minutes. This means that the cost of producing a single 
test varied depending upon the nature of the request and the number of 
requests processed at a single time. On the basis of $100 per computer 
processing hour, this means that if seven tests were processed in 1 
hour and 45 minutes the average cost per test was $25, when four tests 
were processed in 2 hours and 20 minutes the average cost per test was 
$58, and when 11 tests were processed in three hours and 51 minutes the 
average cost per test was $35. Although all members of the computer staff 
considered these costs per test to be extremely high, they noted that the 
cost per test could be reduced markedly if the number of requests from 
teachers was greatly increased. 

Table 7 describes the amount of computer time and the associated cost 
involved in producing computer printouts for teachers. These data were 
collected from the records of a 12-week period in the spring of 1970. 

Another aspect of the operation of the system at the Computer Center 
concerns the turn-around time between initial teacher request and delivery 
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Table 7 

Summary of Time/Cost For Initial 
Requests and Test Print-outs 

Initial Requests Ditto Masters 

Total Aver. Aver. Total Aver. 

Number Computer Time Cost Number Computer Time 

of Time Per Per of Time Per U 

Week Requests (min) Request Request* Requests (min) Request F r^ 
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13 


7 


58 


280 


5 


8 


1 


15 


13 


8 


8 
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14 
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13 








12 
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52 


17 


28 









* This cost is based on a rate of $100 per 60 minutes of computer time. 
It does not include salaries of personnel to operate the Center. 



of the material to the teachers* Because batch-processing was much less 
expensive than processing individual test requests as they were received, 
the requests were usually processed at the end of each day. This meant 
that within 24 hours from the time a request was received the printout 
from the computer was placed in the mails for delivery to the teacher. 
On occasion a slow-up or a delay in the mail delivery system prevented 
the teachers from receiving the computer printouts within a time span of 
two days from their initial request. One additional factor contributed 
to the delay of delivery of the requested materials to the teachers. At 
the end of each month the computer system was completely devoted to the 
school district payroll department. During those days COMBAT requests 
could not be processed. This meant that on occasion there was a delay 
of from five to six days from the initial teacher request to the day 
she actually received the computer printout material. This end of the 
month delay in turn-around time is a result of priorities set at the 
operation level for the total Computer Center. It is not an inherent 
weakness in the design of the system itself. Rather, it represents a 
practical problem in the operation of a computer system that must inter- 
face with a number of other systems at a computer center. 

From this limited examination of the computer system for the COMBAT 
project, it can be seen that a major problem with the utility of the 
system is associated with the extremely high cost for producing a list 
of items requested by a teacher. Although the system could operate on 
a much less expensive set of computer hardware, the existing system 
should be examined much more carefully for alternative procedures for 
reducing the cost of proceasing a request. 

The teacher users* questionnaire contained two structured response 



items and one open-ended item relating to the teachers' acceptance or 
reaction to the material contained on the computer printouts. When 
asked, "What changes in the print-out format for objectives and test 
items could you suggest that would make it (the print-out) more accept- 
able?" Only 15 of the 42 respondents commented. Four teachers indicates 
that test items were too scattered on the dittos. One teacher wanted 
space left for answers for essay questions. One teacher wanted objectivrs 
and test items grouped by subject on the first request and another suggest, 
omitting code numbers. The majority of the remaining comments were relate 
to the content of the items. Three elementary school teachers thought 
the wording of the tests was at too high a level. Although these comments 
constituted a very weak data base, the most frequent volunteered comment:, 
related to the space used on the test dittos. During interviews with the 
developmental staff, two staff members reported teachers had commented 
that the test questions took up too much space on the dittos and that s*. 
school principals did not want teachers to use the computer produced 
dittos because they used up too much paper. 

Teacher responses to the two structured to response questions re- 
lating to the format of the print-outs are included in Table 8. When 
asked to rate the test dittos for "economical use of space on page" only 
24 of the 42 respondents made a rating. Twelve of the 24 rated the 
dittos as high or very high and seven rated the dittos as low or very low. 
These teacher ratings are not totally consistent with the implication 
that the test dittos waste space on the page. However, it should be noted 
that teachers tended to .rate that scale lower than any of the other scales 
reported in Table 8. These scales dealt with the ease of reading the 
ditto, whether the dittos made an adequate number of copies, whether the 



Table 8 

Teacher Rating of Final Test Print-outs 



Rate the test dittos 
you received on the 
following criteria: 
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Rate the form or appearance 
of the final test on the 
following criteria: 
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dittos were compatible with the copy machines at the various schools » 
readability. Item language used, the form of a question, ease of scoring, 
and clarity of symbols. Although the data contained In Table 8 Is limited 
to teacher ratings of the teat dittos, there is an indication that teachers 
reacted favorably toward the printouts they received from the Computer 
Center* The high percentages of nonrespondents to these items was due to 
the fact that the majority of teachers who made requests from the Computer 
Center never made a second request for a test ditto, 

A number of items were included on the general teacher questionnaire 
and the teacher tjsers* questionniare to assess the teacher's perception 
of the adequacy of the telephone-mail communication system between the 
participating teachers and the Computer Center, The general teacher 
questionnaire asked, "Uhere is the nearest phone, at school, that you 
can use to call the COMBAT center?** Of the 472 respondents, 44 percent 
indicated that the nearest phone was in the school secretary's office, 
28 percent indicated the teachers' lounge, a total of 17 percent indicated 
either in their office or the departmental office, 7 percent indicated 
the principal's office, only 1 percent indicated a public pay phone, and 
the remaining 3 percent either failed to respond or specified another 
location. When asWed, '*How easily available is this phone?", 32 percent 
of the same teachers indicated that it was easily available, 36 percent 
indicated it was fairly convenient, 15 percent indicated it was moderately 
inconvenient, 14 percent indicated it was inconvenient, only 1 percent 
indicated that the phone was unavailable, and the remaining 3 percent 
failed to respond. Although these responses show that some teachers may 
have a problem in obtaining access to the telephone before calling the 
COMBAT center, the majority of teachers have access to a phone either in 



the teachers* lounge or in the secretary's office that Is easily avail- 
able or at least fairly convenient. 

Interviews with the COMBAT staff suggested that no formal procedure 
was used to determine the number of days that should elapse between the 
time the teacher makes a request and receives the computer print-out 
materials* Although the developmental staff did survey the teacher users* 
at one time during the project to determine how many days were elapsing 
between the time they requested and received a computer print-out, it was 
felt this question should be explored in more detail* The survey by the 
computer staff found that most teachers received their print-outs within 
two to three days from their initial request and that most of the teacher 
users* seemed satisfied with this length of service* Teachers who re- 
sponded to the general questionnaire and teacher users* questionnaire 
were asked, "In order to make the testing service most effective, what 
would be the ideal and maxlmm number of school days between the day you 
order a list of test items and the day you receive the list?" Table 9 
contains the tabulation of the teacher responses to both of the items on 
the teacher users* questionnaire and the general questionnaire* By 
examining the table it can be seen that the majority of teachers in both 
samples selected the ideal numbers of days as either one, two or three* 
Approximately 20 percent of both of the samples chose "one day** as the 
ideal number of days, while approximately 30 percent of both samples 
indicated three days as the ideal number of days. Twenty-eight percent 
of the teachers responding to the general questionnaire and 36 percent of 
the teachers responding to the users* questionnaire identified the ideal 
number of days as two. This information verifies that the developmental 
staff and the computer staff were accurate in their initial estimates of 



the number of days that should elapse between the time a teacher makes a 
request and the time the computer prlnt**out were In the teachers* hands. 
The data In Table 9 also Indicate there was very close agreement between 
the two samples of teachers In their response to the maximum numbers of 
days that should elapse between the requesting and receiving of a computer 
print-out. Nearly half of the teachers In both samples indicated that it 
should not exceed four to five days. Slightly less than 20 percent of the 
teachers felt that six or seven days could elapse and another approximately 
20 percent felt that it should not exceed two to three days. Teacher users* 
were also asked whether or not the COMBAT center always sent back the test 
items they selected for a final test and how long it took in elapsed school 
days for them to receive the list of test items. The responses to these 
questions are shown in Table 10. Although several did not respond, probably 
because they did not request a test ditto, only four out of th€^<27 who 
did respond indicated the Center did not return the test items they had 
selected. This indicates that there were some problems in returning the 
proper items to teachers, but that in the vast majority of castas the Compute - 
Center was successful in getting the right items to the teacher « The 
majority of teachers received the print-outs from the Computer Center 
within three to four days. Only six of the 34 teachers respondii^ to 
this item said it took five or more days to receive the print-outs from 
the Computer Center. Althot^h the Center was not always successful in 
holding to the intended turn-around time of two days, the tum*-around 
time exceeded four days in less than 15 percent of the requests. 

The teacher users were also asked to make one positive and one neg- 
ative comment about procedures for obtaining objectives, lists of test 
items, and test masters. Twenty-eight teachers made a positive comment. 
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Table 9 



Desired Elapsed Time Between Initial 
Request and Receipt of Test 

Question: In order to make a testing service most effective, what would 
be the ideal and maximum number of school days between the day 
you order a list of te^t items and the day you receive the list? 

General Questionnaire; 



ideal number 
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N.R. 


maximum number 


4% 


18% 


44% 


16% 


7% 


9% 


of days 


18 


85 


211 


77 


36 


45 


0-1 


2-3 


4-5 


6-7 


8 or 


N.R. 












more 





Users ^ Questionnaire; 



ideal number 
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Table 10 
Item/Test Selection and Receipt 

Did the COMBAT center always 55% 10% 33% 
send back the test Items you 23 4 14 

selected for the final test? yes no N.R. 



How long did it take (in 24% 46% 10% 5% 17% 

elapsed school days) for 10 19 4 2 7 

you to receive the list 1-2 3-4 5-6 6-7 N.R. 



Seven teachers said the service was fast, four indicated that the people 
at the COMBAT center were extremely cooperative, and two others indicated 
that it was easy to phone in. The remaining 15 comments did not relate 
to procedures for obtaining the items. Twenty-eight teachers made negative 
comments. Only six of the negative comments pertained to the procedures 
for obtaining the items. Two teachers said it took too much time, two 
said they did not plan for tests far enough in advance, and one teacher 
said that the use required very careful wording and another said that the 
phone*- in hours should be other than school time. The other comments 
related to item content or to depth and breadth of items. In general it 
can be stated that the teachers were moderately to generally favorable 
in their reactions to the telephone-mail communications system and to 
the format of the print-outs they received from the Computer Center. 

The major hindrance in the present system rests with the time It takes 
to generate the list of items for teachers and with the requirement that 
the present system utilize the total computer capability as it processes 
the teacher requests. This latter requirement of the present system has 
resulted in the inability for the Computer Center to process requests very 
quickly when it had a heavy load of other activities such as the payroll 
department work. It should be further noted that the Computer Center 
personnel indicated that batch processing is more efficient but that the 
batch processing can best handle approximately ten requests per batch. 
Based on information provided by the Computer Center, a batch processing 
of ten requests would require approximately two hours of computer running 
time depending upon the complexity of the request. This means the maximum 
number of requests that could be processed by the computer per day would 
peak out at approximately 120. In order to reach that maximum number, the 



present system would utilize the computer for a full twenty-four hour 
period. It can be easily determined on this basis that as the computer 
file grows to three or four times its present size, the maximum number 
of requests that could be processed per day would be somewhat reduced. 
At the same time, as the capacity of the file increases it can be anti-- 
cipated that the number of requests per day would increase. It there- 
fore becomes apparent that in order to make the present COMBAT computer- 
based testing program maximumly efficient, an alternative procedure for 
retrieving the items from the files should be explored. 

Objective 4 ; To conduct the necessary teacher in-^servlce training 
required to insure optimum utilization of the test development center . 
The developmental staff of the COMBAT project conducted several activi- 
ties that were designed to serve as teacher in^ervlce programs. Among 
these activities were the initial COMBAT test tryouts in the fall of 
1968 with 34 teachers from the fourth through the sixth grades. Depart- 
ment of Continuing Education classes in Measurement of Educational Ob- 
jectives during the Fall and Winter of 1968, individual school building 
or district level orientation sessions, and orientation^ dinners for a 
group of teacher advisory teams. The initial teacher workshops during 
the Summer o f 1968 could also be Included In this category of activity. 

The record of teacher users maintained by the Computer Center 
indicated that by late spring, 1970, 470 teachers had requested lists 
of test items from the COMBAT center. These teachers had made a total 
of 1,144 requests of which 922 were initial requests and 222 second 
requests. As defined earlier, a first request was an initial contact 
by a teacher with the Con5)uter Center for a list of test items or ob- 
< jectives, while a second request was classified as a second contact by 
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the teacher relating to an earlier first request. On the second request 
the teachers ustjally ask for a test ditto to be prepared. Table 11 
contains a two-^ay classification of the number of teachers who made 
first and second requests. As seen In the table, 215 teachers made one 
first request and no second request, 49 teachers made two first requests 
but no second request, 62 teachers made one first request and one second 
request, 22 teachers made two first requests and two second requests. 
As can be seen by the marginal totals, approximately 70 percent of the 
teachers making requests never made a second request. This data con- 
firms the comments made by the project staff that teachers made light 
use of the system. 

The record of teachers using the system was cross-referenced with 
the names of the teachers who participated in each of the training 
activities. Of the 15 teachers who participated in the initial Summer 
Workshop for writing objectives in 1968, only one of these teachers was 
recorded as having made a request from the COMBAT center. This teacher 
made 14 first requests and 3 second requests. 0£ the 34 teachers who 
participated in the initial COMBAT tryouts in the Fall of 1968, only 19 
were reported as having made requests to the Computer Center. Of these 
19, six made a single first request and four made two first requests. 
Eight of the nine remaining teachers made from four through seven request^ 
and one was recorded as having made a total of 29 first requests. All but 
three of these 19 teachers made at least one second request; eight made a 
single second request and the remaining eight made from two through six 
second requests. One hundred twenty teachers were recorded as having 
participated in the DCE classes. Fifty-four of these teachers had no 
record of making a request at the Computer Center. However, 24 of these 
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Tabulation of Teacher Requests 
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54 individuals were recorded as having attended the same class. Of the 
56 teachers from the DCE classes who were recorded as making at least 
one request, 34 made a single first request and no second request. Six- 
teen of these teachers made a single second request and one made seven 
second requests. One teacher was recorded as Having made 8 first request • 
and 7 second requests. Seventeen teachers made from two through five 
first requests with the majority of these making a single second request. 

The teacher advisory-team dinner meetings were designed to include 
a number of administrative staff and counselors from the school districts 
serviced by the COMBAT system. Of the 174 individuals invited to partic- 
ipate in the dinner orientations, 79 were In nonteaching positions. 
Eight of these 79 nonteaching professional personnel made at least one 
request to the COMBAT center*. Of the 95 teaching personnel attending 
these sessiohs, 72 were not recorded as having made a request of the 
COMBAT system. Of the 23 teachers that made a request, 16 made a singltc 
request. Of the total 174 personnel attending these sessions, 15 made 
more^ than one request. Of these, two made one first and one second 
request, three made two first requests and no second request, two made 
two first and two second requests, four made four first and no second 
request, one made four first requests and one second request, three made 
five to seven first requests without a second request, and one was re- 
corded as having made 14 first requests and three second requests. There 
are no records that allow the ^^e termination of the number of requests 
made by the personnel who attended the school district or building orien- 
tations sessions conducted by th^ COMBAT staff. 

On the whole, the record data indicate that in-service programs 
designed to encourage teachers to use the system were not successful in 



producing a large number of teacher users* of the COMBAT system* This 
judgment is based on the evidence that a large number of teachers who 
participated in the program did not use the system or used it only to 
a very limited extent. Items were contained on the interview guides 
and questionnaires to take a closer look at the objective. 

The members of the Advisory Board were asked the question, "Do you 
think that the procedures used to insert the COMBAT system into the 
on-going programs of the participating schools (such as: workshops, 
in-service programs, and teacher orientation meetings) were the best 
possible?" All of the Bo ^rd members responded "yes," with some qualifi- 
cations. They all felt that the problem of orientation and getting 
teachers to use the system existed throughout the project, but that all 
of the staff members worked very hard to solve this problem^ Each of 
the members of the Board indicated the Advisory Board had been consist- 
ently involved throughout the project in attempting to identify procedures 
to Improve the teacher utilization rate. When the Board members were 
asked, "What other strategies might have been employed to insert the 
COMBAT system into the schools?" a variety cf reactions were obtained. 
Three members of the Board indicated they felt the project should have 
worked more closely with the curriculum directors in the school districts 
rather than with the testing representatives. Three of the members of 
the Board stressed that they felt the project should have been more fully 
developed before an attempt was made to encourage the teachers to use the 
system. One member of the Board indicated the school principals should 
have been more involved in the project operations and at least one teacher 
from each o. ,ae schools should have been actively involved in the project 
development, suggested these procedures would have helped to encr urage 
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the individual teachers In the buildings and the school districts to use 
the system to a greater extent. One member of the Board said the stress 
should have been placed on the development of individualized instruction 
rather than on an addition to the testing program*. 

The questionnaire mailed to the district representatives to the 
MATPB organization contained two items related to the in--service program* 
The first item asked testing representatives to rate the effectiveness oi 
each of the in-service programs. Of the 14 responses obtained, the 
majority tended to rate the effectiveness of the in-service programs as 
average. However, the ratings differed somewhat for each cf the activities,. 
The testing representatives were asked to rate each in-eervlce program as 
to th<^ degree of effectiveness on the basis of a five-point scale: very 
low, low, average, high, very high. For the Summer Workshop, 1968, five 
rated the program as average in effectiveness, four rated it as high, 
two rated it low or ver>' low, and three failed to respond. The Fall, 
1968, initial experiuental trials in social studies for grades four 
through six were rated average by six representatives, high by one, low 
by one, and ^±x representatives failed to respond. Teacher classes on 
Measuring Educational Objectives tended to be rated highest by the district 
representatives. Six rated those as being highly effective, three rated 
it as average, two rated it as low, and three failed to respond. Seven 
of the fourteen rated the teacher advisory dinners as average in effec- 
tiveness, two rated them very high, one rated high, two rated them low, 
and two failed to respond. The distrl :t and building level teacher 
orientation sessions were rated average by five representatives, three 
rated high, two rated low, and two rated very low, and two failed to 
respond. The representatives were asked to respond to the question. 
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"What other strategies might have been employed to increase the use of 
the COMBAT system in participating schools?" Nine of the fourteen 
representatives responded to this open-ended item. Four of the responses 
suggested these people felt the project should have used school district 
personnel, with whom teachers were more familiar, as the basic source 
of information about COMBAT* These personnel would include curriculum 
directors and district testing personnel. Two of the comments indicated 
the representatives felt that the project should have been developed to 
a full extent before it was made available to teachers. The other three 
comments were as follows: "Make available, or circulate, a very accurate 
and uninflated listing or print-out of subject areas in which objectives 
and items are available", "Some methods whereby teachers will more readily 
try the program", "SenJ teams to school buildings to sell COMBAT." 

Both the teacher Users* Questionnaire and the teachers* General 
Questionnaire contained an item designed to determine what proportion 
of the teacher users and the general teacher population participated 
in each of the in-service activities. Table 12 contains a tabulation 
of the responses. As can be seen in the table, the Fall and Winter 
D.C.E. classes and the building or district level orientation meetings 
effected the greatest proportion of teacher users. Only a very small 
number of the teacher users reported involvement in any of the other in- 
service activities. The general teacher questionnaire indicated that the 
building or district level orientation meetings were able to reach approx- 
imately 9 percent of the teachers. The other activities affected only 
very small percentages of the total teacher population* 

When the teacher users were asked, "How do you rate your familiarity 
with the COMBAT system?" approximately 30 percent responded "near zero 
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Table 10 



Attendance of In-Service Programs 



Teacher Users ' : Check each of the following activities in which 
you participated. 



Summer 1968 
Workshop 

Fall 1968 
Initial Trials 

Fall & Winter 
68-69 D.C.E. 
Workshops 

Building or District 
Level Orientation 
Meetings 

COMBAT Advisory 
Team Dinner, 1969 



No. 



10 



12 



% 

3% 
5% 

24% 

29% 
8% 



General Questionnaire : Check each of the following activities in 

which you participated. 



Summer 1968 
Workshop 

Fall 1968 
Initial Trials 

Fall & Winter 
68-69 D.C.E. 
Workshops 

Building or District 
Level Orientation 
Meetings 

COMBAT Advisory 
Team Dinner, 1969 



No. 
6 

20 

14 

43 
10 



% 

1% 
4% 

3% 

9% 
2% 
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or low," 30 percent responded "barely enough to use it," and approxi- 
mately 36 percent responded "moderate," while only 3 percent responded 
"high." Part of this fairly low rating of their familiarity with the 
system may be due to the low rate of use by the teachers and/or a time 
lag from the last time they used the system to the date of the question- 
naire. Slightly over 50 percent of these teachers indicated they have 
not used the system since the 1968-69 school year. IWenty-four percent 
indicated they had not used the system since the end of December 1969, 
while 14 percent of the teacher users indicated they hadn' t used the 
system since January, 1970. Eight percent of the teacher users did not 
indicate when they had last used the system. 

The general teachers » questionnaire contained two items that 
attempted to detemine the approximate proportion of teachers in the 
target population that have used rhe COMBAT system. When asked if they 
had ever used the COMBAT system, 13 percent of the respondents indicated 
"yes", 85 percent indicated "no", with 2 percent not responding. Eleven 
percent of the total respondents indicated they had used the system once 
or twice, while the remaining two percent reported they had used the 
system more than twice. When these same teachers were asked whether or 
not they had ever used the test items from the COMBAT system, 11 percent 
responded "yes", 86 percent "no", with 3 percent not responding. Nearly 
70 percent of those that reported they had used the system also signified 
they used the items once or twice, while the remaining teachers indicated 
that they had used the items more than twice. 

In addition to the in-service activities conducted by the develop- 
mental staff, several procedures were used to distribute information 
about the COMBAT system. These methods included the distribution of 
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brochures to individual school buildings through the cooperation of the 
County I.E.D. agencies, the distribution of telephone stickers advertising 
COMBAT, the production and distribution of a slide-tape presentation, 
distribution of matchbooks advertising COMBAT, the distribution of 
COMBAT puzzles, a presentation about COMBAT at the 1969 Oregon Edu- 
cation Association State Convention, and an article about COMBAT in the 
December, 1970, issue of Oregon Education — a publication of the Oregon 
Education Association, Based on the responses to the teacher question* 
naire, the COMBAT brochure was by far the most effective of these pro- 
cedures. Fifty-two percent of the teacher v^ers reported that they had 
seen a copy of the brochure, while 35 percent of the general teacher 
population indicated they had seen a copy of the brochure. Twenty-seven 
percent of the teacher users and 15 percent of the respondents to the 
general questionnaire said they had seen the article in Oregon Education . 
Seventeen percent of the teacher users and 7 percent of the respondents 
to the general questionnaire had seen the telephone stickers, while 14 
percent of the teacher users and 9 percent of the respondents to the 
general questionnaire had seen the slide-tape presentation. The remaining 
activities reached only small percentages of the teachers. Nineteen percent 
of the teacher users and 48 percent of the general teacher population had 
seen none of the documents or presentations listed above. 

The testing representatives from the districts were asked to rate 
the effectiveness of each of these methods of dissemination of information. 
The dissemination tactic that received the highest rating by the represen- 
tatives was the COMBAT slide-tape presentation. Five of the respondents 
rated this as high, four rated it as average, two rated it low or very 
low, with three not responding. Distribution of brochures was rated as 



high by two members of this group, while seven rated it average and 
three rated it low, with two not responding. The telephone stickers 
were rated as average by six representatives, four rated it low, and four 
failed to respond. The journal article was rated as very high by one 
representative, high by a second, average by seven, low by two, with 
three not responding. Distribution of the matchbooks was rated as 
average by four representatives, low by four, very low by two, with 
four not responding. The distribution of COMBAT puzzles was rated as 
average by four, low by four, very low by one, with five not responding. 
The COMBAT presentation at the State Education Association Convention 
was rated high by one representative, average by three, low by five, 
and very low by three, with two not responding. 

The general teacher questionmiire also asked the question, "If you 
intended at this moment to call the COMBAT system for any reasons, how 
easily could you locate the telephone number?" Thirty-one percent of 
the respondents indicated they hadn't the "slightest idea who might 
have the number," while 50 percent reported they know who should have 
it or who might have it. Four percent of the respondents indicated 
they "thought they knew where they wrote it down,'* while 14 percent 
indicated they knew "exactly where to locate the number." 

Although the developmental staff conducted a wide range of activities 
aimed at distributing information about the COMBAT system and encouraging 
teachers to use the system, the data indicate they were only moderately 
successful in this effort. There is also an indication in the data that 
dissemination by personnel outside of the individual school districts 
lack efficiency. Wide-spread distribution of information relevant to 
the system is probably highly dependent upon the cooperation of school 
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district personnel that have day-to-day contact with teachers* Such 
procedures probably require the training of specialized school building 
personnel to a degree that would make them equivalent to building 
consultants. Such procedures would require a considerable amount of 
volunteered personnel time by individual districts or fairly large 
amounts of special funded resources to pay each of these building repre- 
sentatives to participate in extensive training sessions. 

Objective 5 : To Improve the overall effectiveness of instruction 
i n all curricular areas and grade levels presented in the elementary and 
secondary schools in the participating districts . There is no evidence 
that the project staff conducted any in-depth surveys of the testing 
patterns used by teachers during the early stages of the project* In 
an attempt to obtain some baseline data from the general teacher popula- 
tion and teachers who have used the COMBAT system, questions relating 
to teachers* testing patterns were Included in both teacher questionnaii. 
Responses to these Items are tabiilated In Tables 13 and 14. The General 
Questionnaire asked, "How often do you test in your classroom?" As indi 
cated by the responses tabulated in Table 13, most teachers responding 
to the general questionnaire stated "as it seems appropriate." Thirty 
percent of the teachers indicated they tested after each major unit, and 
23 percent indicated they tested after each instructional unit. There 
was some variation on testing patterns depending upon grade level. The 
major differences In the trends by grade level were that a large propor- 
tion of secondary teachers indicated they tested after each major instruc- 
tional unit. The reasons for testing are also listed in Table 13. As 
shown, a large proportion of the teachers used tests for grading and 
reporting student progress ^ to diagnose the student weaknesses and to 



Table 13 



Teacher Testing Patterns 



How often do you test in your classroom? (Check each appropriate category.) 

No. % 

After each ins true- 112 23% 
tional unit. 

After each MAJOR 140 30% 
instructional unit. 

Daily 10 2% 

Weekly 82 17% 

Monthly 17 3% 

« 

Quarterly 6 1% 

As it seems 329 70% 
appropriate . 

Each grading period. 25 5% 



Why do you give tests? 



No. % 



To plan instruction. 179 38% 

For grading and reporting 302 63% 
student progress. 

To monitor student 250 53% 

progress. 

To diagnose student 292 61% 

weaknesses . 

To provide guidance for 185 39% 

students* individual study. 

To motivate study. 185 39% 
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Table 13 (continued) 



Approximately how often to you test for each of the following purposes 



Pretest 



13% 
61 
Never 



35% 
167 ' 
1-3 
times /yr 



13% 
59 



4-6 
times /yr 



11% 
50 
1-2 
times /mo 



13% 
61 



3 or more 
times /mo 



Posttest 



3% 
17 



Never 



13% 
60 



20% 
96 



1-3 4-6 
times /yr times /yr 



28% 
132 



1-2 
times/mo 



19% 
93 



3 or more 
times /mo 



Diagnostic 
Test 



5% 
26 



Never 



38% 
181 
1-3 
times /yr 



17% 
88 



4-6 
times /yr 



15% 
70 



1-2 
times /mo 



7% 
33 



3 or more 
times / mo 



To check on instruc- 
tional effectiveness 
but not to grade 
students . 



10% 
45 
Never 



23% 
111 
1-3 
times /yr 



19% 
92 



4-6 
times /yr 



17% 
80 



1-2 
times /mo 



13% 
61 

3 or more 

times/mo 



Table 14 



Teacher Users' Testing Patterns 



Approximately how often do 

you test, in your own program, 

for each of the following purposes? 





8% 


27% 


19% 


14% 


5% 


29% 


(a) Pretest 


3 


11 


8 


6 


2 


12 




Never 


1-3 


4-6 


1-2 


3 or more 


N.R. 






times /yr 


times/yr 


times /mo 


times /mo 






0% 


10% 


33% 


19% 


14% 


24% 


(b) Posttest 


0 


4 


14 


8 


6 


10 




Never 


1-3 


4-6 


1-2 


3 or more 


N.R. 








^ "f mo c / V "1* 

wXillCJ / Y i. 


uxmco / mu 


^ A vwiA^ finite 

txmes / mo 






5% 


29% 


14% 


19% 


8% 


27% 


(c) Diagnostic Test 


2 


12 


6 


8 


3 


11 




Never 




H— 0 


X— z 


3 or more 


M D 

iN • K. 






txmes /y r 


txmes/y X 


times /mo 


times /mo 




(d) To check on 


5% 


19% 


22% 


14% 


8% 


33% 


instructional effec- 


2 


o 




0 






tiveness but not to 


Never 




H— 0 


X z 


o or more 




grade students. 




times /yr 


times/yr 


times /mo 


times /mo 




How many times have you used 












the COMBAT test items 


for: 














27% 


XV/ /e 


On 








(a) a pretest 


11 


4 


3 


2 


3 


19 




0 


X 


0 




4 or more 


IN • Ka 




24% 


8% 


12% 


17% 


5% 


55% 


(b) a posttest 


10 


3 


5 


7 


2 


23 




0 


1 


2 


3 


4 or more 


N.R. 




33% 


17% 


5% 


5% 


0% 


41% 


(c) a diagnostic 


14 


7 


2 


2 


0 


17 


test 


0 


1 


2 . 


3 


4 or more 


N.R. 


(d) a test designed 


38% 


3% 


0% 


3% 


0% 


57% 


for one student 


16 


1 


0 


1 


0 


24 


or a small group 


0 


1 


2 


3' 


4 or more 


N.R. 


of students. 














Approximately \^at % 


41% 


24% 


14% 


5% 


0% 


17% 


of your student grad- 


17 


10 


6 


2 


0 


7 


ing or evaluation is 


0-20% 


21-40% 


41-60% 


61-80% 


81-100% 


N.R. 



based on tests composed 
of items like these stored 
in the COMBAT system? 
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monitor student progress. Approximately 40 percent of the teachers 
reported they tested to plan instruction, to provide guidance for 
student individual study, and to motivate study. There were also some 
shifts in the patterns of reasons for testing at the different grade 
levels. Slightly greater proportions of elementary teachers used tests 
for planning instruction, whereas a somewhat greater proportion of 
secondary teachers used the test to motivate study. 

Approximately 85 percent of these teachers indicated they "mainly 
obtained test items from the information contained in their instructional 
materials^ such as textbooks, films, and recordings.** Of this majority, 

approximately 75 percent wrote their own test items, whereas one quarter / 
used items prepared by textbook publishers. Approximately 80 percent of 
the teachers in the general population indicated they did not use any 
special references in constructing test items. The last item in Table 
13 and the first item in Table 14 are responses to the same question on 
the two teacher questionnaires. Although the percentages differ in the 
two sets of responses to the question of how often teachers pretest, 
posttest, use diagnostic tests, or test to check on instructional effec- 
tiveness, there are few consistent trends that can be observed by 
examining the two tables. 

It was felt that these two tables may provide some clue as to 
whether or not teachers who had been exposed to the COMBAT system had 
changed their testing patterns. However, the evidence is not in the 
data. Table 14 indicates that teachers used the COMBAT system for post- 
tests much more frequently than for any other purpose. However, the first' 
tabulation in Table 14 indicates that the same pattern existed in the 
teachers general use for tests. A further examination of the first two 
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ite^ .n Table 14 shows that teachers used COMBAT items less often for 
d-'? gnostic purposes than for posttest purposes as compared to their 
general ratio of diagnostic tests to posttests. The last item in Table 
14 shows that slightly over 40 percent of the teachers reported they 
could use the items contained in the COMBAT system for approximately zero 
to 20 oercent of their student grading or evaluation practices. Approxi- 
mately o..^- fourth of the teacher users said they could use COMBAT itemr> 
for 20 to 40 percent for their total testing program. Fourteen percent 
indicated that they could use COMBAT icems for 41 to 60 percent of their 
testing program, whereas five percent indicated that they could use COMBAT 
items for 61 to 30 percent of their testing program^ Seventeen percent 
of this group of teachers failed to respond. 

In order to get a more direct measure of the impact of the COMBAT 
system on classroom instruction, teacher users were asked the question, 
"How did the COMBAT system change your classroom instruction in each of 
the following are's: frequency of testing, frequency of pretest, planning 
tests related to specific objectives, planning Instruction for specific 
objectives, use of tests in new ways, use of objectives in new ways, 
time required .'or test construction?" Approximately 30 percent of the 
respondents failed to answer this question. Of those that responded, 
approximately 80 percent indicated there was no change in the frequency 
of testing. The remaining 20 percent were about equally split between 
"more often" and "less often." Approximately two-thirds of those respond- 
ing reported the "frequency of pretest" was unchanged as were "using tests 
in new ways," and "using objectives in new ways." The remaining one-third 
indicated they "more often" used pretests, used tests in new ways, and 
used objectives in new ways. Approximately 60 percent of the teachers who 
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responded to this question Indicated that they "planned Instruction for 
specific objectives" more often or much more often, whereas 40 percent 
Indicated no change. All but 7 of the teachers responded to the portion 
of the Item that asked them how the COMBAT system effected the "time 
required for test construction," Of the 35 responding, approximately 20 
percent luuicated It took "more time," 20 percent Indicated "no change,'* 
and 20 percent Indicated "less time," while 10 percent Indicated It took 
"much more time," another 10 percent indicated that it "took much less 
time." On the basis of these responses, there was some positive impact 
on the instructional practices of the teachers using the COMBAT system. 
Although this effect was not great, the trend was in the positive 
direction. 

The questionnaire distributed to the district testing representa-*- 
tives asked the question, "What have you actually observed about teachers 
and/or classrooms that illustrates the Impact of the present COMBAT 
system?" Ten of the 14 who returned the questionnaire said they had not 
observed any direct impact of the COMBAT system on the classrooms. 
However, some of these representatives qualified their answers by stating 
that teachers opinions varied as to the feasibility of the program and 
when properly used they could see value in the COMBAT system. Two said 
they had not been involved with the COMBAT project long enough to actually' 
observe direct impact. One reported he had heard teachers state they 
were more aware of their own objectives and now have more direction in 
their teaching. Another indicated that those teachers he had talked to, 
who had used the system, appreciated the service. Another testing repre- 
sentative indicated he had found teachers who had used the system highly 
enthusiastic for what they obtained. One representative reported the 
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observation of an instance of ona teacher using the system to test fourth 
grade students on Northwest Indians. He further commented that she 
reported it as a "good test and was plee.t u with the accompanying map 
materials." One representative said "teachers who had used the system 
and found a lack of materials were frustrated or disappointed" with the 
COMBAT project. 

When the representatives were asked to comment on the most critical 
project decisions and how these decisions affected the project, the most 
common reply was related to the decision to expand the CO^^BAT system to 
include all grade levels in four curricular areas. In general, the 
representatives felt this decision encouraged tescher use of the system 
before it was adequately developed. The result was teacher frustration and 
dissatisfaction with the materials they obtained from the COMBAT service. 
On the positive side, one representative reported the decision to ask 
teachers for their objectives and items produced wider and more frequent 
use of the system. When asked if they agreed with the majority of the 
project decisions relating to its operation, 6 of the 14 representatives 
indicated "yes," two indicated "no," and six declined to respond. Here 
again, the majority of the respondents commenting on these decisions 
indicated they felt the project expanded much too rapidly. They indicated 
the rapid expansion caused the project to lose face with teachers who 
attempted to renuest items for which there were few items stored in tha 
pool. 

Members of the Advisory Board were asked, "What are some of the 
ways you thought the system would be manifested in the classroom?" One 
individual reported the system would force teachers to identify "what 
they were trying to accomplish." Two reported the system provides the 
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teachers with pretest knowledge, objectives for planning a unit based 
on pretest, and posttest knowledce. These representatives also mentioned 
the system could be used for diagnostic testing. Three representatives 
said they could visualize the system providing the needed curriculum 
for individualization both on a one-to-one basis and on a group basis* 
Two individuals indicated the use of the test Items and then the possible 
addition of objectives after familiarization with the system. 

When members of the Advisory Board were asked, "What manifestations 
of the system have you actually observed In the classrooms served by the 
system?" all of the Advisory Board reported only limited or no direct 
observations, not enough to^make any definite statements about changes 
caused by COMBAT. When asked, '*What changes should be made in the 
operating procedures of the system so that It might attain the potentlalr 
you described?" two of the Advisory Board reported they would stress a 
complete workable file in only one area where the teachers could rely on 
the system to run smoothly from the very beginning. Two of the Advisory 
Board indicated that anyone designing a similar system should have 
adequate financial support. Other recommendations were "start from the 
bottom, the teachers level, the move to the district level, etc.'* and 
"have a preconceived taxonomy where materials may be collected to fill 
the gaps in the specified areas and the computer system should be built 
to accommod*»we random access to Improve turn-around time." 

When the Advisory Board was asked, "What were the most critical 
project decisions as you perceive them, and how did they affect the 
project?" The following replies were obtained: "The decisior to involve 
the testing people Instead of curriculum people was a mistake. If 
curriculum people would have been involved, there would have been a more 
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solid base and closer contact with teachers/' "The decision on what 
area the initial item pool would cover and the program for input and output 
of data. The real problem was the COMBAT project should have been designed 
for five years rather than three," "The point at which to make the project 
operational was a critical decision," "We should have stayed with the 
fourth, fifth and sixth grade social studies and not expanded to other 
areas until enough money was budgeted to supply teachers for writing 
objectives and items," "The two critical decisions were the expansion 
of the item file and the use of teachers* written test items in the item 
file." Overall the Advisory Board seemed to be most concerned about the 
project's decision to expand the item pool to all the grade levels in 
four curricular areas before any one area was completely developed. 

All in all there is little evidence that the COMBAT project had 
any dramatic impact on the classroom ins*truction in the schools affected 
by the project. However, there was no baseline information from which 
to make comparisons and no project long range systematic effort to collect 
data for this objective. There is some evidence indicating that the 
project was making a slight increase in the number of times teachers used 
tests and in the way in which they used tests. However, the move in 
that direction was not great. Teachers who used the system reported the 
greatest change occurred in the frequency to which they taught to specific 

m 

objectives. This evidence suggests that when the system is completely 
developed there is a good potential for a computer-based testing program, 
including files of behavioral objectives, to significantly effect class- 
room instruction. However, before this can occur, the item files must be 
more thoroughly developed than was possible in the three years of the 
current COMBAT project. 
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Objective 6 : T o conduct the necessary Informati o n disse mi nation 
activities which vlll make this system a model for similar centers to 
be located throughout the United States , This objective can be Inter- 
preted In at least two ways. One Interpretation relates to the dissemina- 
tion of information within the area affected by the project Itself. The 
other Interpretation concerns the dissemination of information to inter- 
ested parties outside of the COMBAT service area. Much of the data 
recorded for Objective 4 relates to the first interpretation. The reader 
Interested in this interpretation of Objective 6 is referred to the 
discussion presented under Objective 4. 

Educators located outside of the Immediate service area had three 
primary sources for Infomatlon about the COMBAT project. These Included 
presentations at national professional association meetings such as the 
annual meeting of the American Educational Research Association, the 
National Council on Measurement in Education, and the National Assoclatlui 
for School Administrators. During late 1969 and early 1970, members of 
the project staff made formal presentations about the COMBAT system to 
each of these groups. Other sources of information were contacts through 
the Title III project office in Washington, D.C. and the distribution of 
newsletters by the project developmental staff. The developmental ^taJ^f 
reported receiving a total of 33 formal requests for information about 
the COMBAT system from individuals out^iUle of the service area. These 
requests came from individual ^chfi.l districts, universities, research 
and developmental centers, as well as from individual Interested people. 
Fittieen of the requests came froii the East Coast states, 8 fron the 
Midwest states, 3 from West Coast states, 2 from countries outside of 
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the United States, and 5 from areas within the state of Oregon not 
serviced by the project. 

During the 1968-69 academic year the developmental staff produced 
and distributed a COMBAT Newsletter bimonthly. As of mid-July, 1969, 
the COMBAT Newsletter mailing list contained slightly over 300 names. 
However, in the 1969--70 academic year only one version of the COMBAT 
Newsletter was produced and distributed. The developmental staff re- 
ported the primary reason for the slow down in production of the News- 
letter was due to a shift in responsibility for publication. During the 
previous year the developmental staff had composed and published the 
Newsletter but during the past academic year, the Multnomah County 
Intermediate School District was responsible for its publication. There 
appeared to be some communication problems between the two agencies in 
carrying out production of this Newsletter. This problem apparently 
arose as a result of shared production responsibility. 

Much of the distribution of information to interested people out- 
side of the immediate service area was limited to exchange of items and 
objectives and to general information about the intents and activities 
of the COMBAT project. On the basis of information obtained by the 
evaluation staff, there appears to be a lack of complete documentation 
of the specific details of the development of the test items and develop- 
ment of the computer system. Part of the explanation for this was a lack 
of staff time to devote to tiiis activity. Because limited resources 
were available to the two technical staffs of the project, the time re- 
quired to completely document all of their activities was not available. 
Lack of sufficient funds for this type of activity is very common with 
projects sponsored by funding agencies. If the dissemination activities 
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of projects similar to COMBAT are to improve, funding agencies will need 
to provide the resources for personnel to con^letely document their 
activities* 

C. OBJECTIVES OF T!IE THIRD YEAR 

In addition to the objectives of the first two years, the third year 
of the project attempted to obtain four goals. These were: a) to in- 
crease the files by at least 20,000 objectives and items in grade levels 
four through twelve in the following subject areas: Mathematics, Science, 
Social Studies, and Language Arts, b) to introduce additional teachers to 
the COMBAT system and encourage regular usage by teachers at all grade 
levels, c) to increase the number of schools actively using the COMBAT 
system, from 79 of the possible 250 elementary schools in the MATPB area, 
to at least 150 elementary schools and at least one-half of the 59 
secondary schools, d) to provide an index of objectives and items listed 
by content areas for the grade and subject matter areas for which materiaLs 
are developed. 

On June 1, 1969, the COMBAT files contained 7,703 items. By June 1, 
1970, the COMBAT files contained 32,163 items. Clearly, the project 
attained its objective of increasing the files by 20,000 items. The 
project exceeded its objective by nearly 5,000 items. 

The record data did not permit a determination of how many of the 
tfitichers, reported as having used the COMBAT system, were added to the 
list of users after June, J969. However, the responses to an item on the 
teacher users* questionnaire indicated approximately 50 percent of the 
teacher users had used the system during the 1969-70 school yesr. Records 
of the number of requests made each month shows there was an increase in 
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the number of requests during the latter months of 1969 • During the 
month of September, 1969, there were a total of 34 first and second 
requests. The number jumped to 170 requests during the month of 
October and to 138 first and second requests during the month of 
November. In November, 1963, there were only 28 first and second requests. 
In December, 1968 and 1969, the number of requests were 91 and 139 re- 
spectively. These comparisons provide some Indication that the orientation 
sessions during October, 1969 helped to increase the number of teachers 
making requests of the COMBAT system. 

By June of 1970, a total of 148 elementary schools and 64 secondary 
schools had one or more teachers request items from the COMBAT center* 
This means the project was successful in its objective to increase the 
number of elementary schools having at least one teacher using the COMBAT 
system from 79 to approximately 150, The project exceeded its objective 
of getting teachers in at least 30 secondary schools to use the system. 
A total of 54 secondary schools were recorded as having at least one 
teacher make a request of the COMBAT center* Of these 54 secondary 
schools, 21 recorded one teacher making a request, 13 recorded two 
teachers, eight recorded three teachers, eight recorded four, five, or 
six teachers, three recorded seven or eight teachers, and one had eleven 
teachers making a request of the COMBAT center* These teachers made a 
total of 280 first requests and 24 second requests for a total of 304 
calls to the COMBAT center* In all, a total of 146 teachers in the 
secondary schools made requests of the COMBAT center for an average of 
slightly over two per teacher* 

During the fall of 1969, the developmental staff prepared an index 
of key words in each subject area for different grade levels. Copies 
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of these indexes were distributed to individual teachers and school 
buildings with the cooperation of the Intermediate Education District 
offices. In response to the question, "Do you have access to the COMBAT 
computer-based test development index for your grade level and subject 
area?" Sixty-two percent of the teacher users indicated "no," 18 percen.' 
indicated "yes," and 20 percert failed to respond. This result demonstrate^ 
that the indexes were not adequately distributed to teachers in the service 
area and the follow-up of the distribution procedures for the indexes was 
not adequate. Because of the small number of teachers having access to 
the indexes, teachers responses to the usefulness, complexity, and com- 
prehensiveness of the index were too small for reporting in this evalua- 
tion. 

In general, the objectives of the third year were met or exceeded 
by the project staff. The pri'nary exception to this general trend was 
the lack of adequate distribution of the index of objectives and items 
to teachers In the area serviced by the project. Although the indexes 
were prepared, they failed to reach the teachers. 

D. PERCEIVED PROJECT POTENTIALS 

Innovative developmental efforts have traditionally been difficult 
to implement in the classrooms of elementary and secondary schools. 
As indicated by the moderate nimiber of teachers that used the COMBAT 
system, this project was no exception to the rule. However, the initial 
success of convincing teachers to use an innovative idea is not always an 
adequate measure of the potential of the innovation. For thesr, reasons, 
the ^valuators attempted to determine the strength and weaknesses of thf 
COMBAT system as perceived by teachers, testing representatives, and the 
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Advisory Board. These same educators were asked for their opinions about 
the appropriate directions for future development of computer-based 
testing programs. 

When teacher users were asked to rank six possible uses of the COMBAT 
system in order of usefulness, there was some confusion in the manner in 
which teachers responded. However, on the basis of the data that were 
obtained, the teacher users ranked "tested the effectiveness of specific 
instructional methods or materials" and "provide feedback for students on 
their current level of learning (achievement)" as the two most important 
potential uses. "Enhance the teacher's role as an instructional manager" 
was ranked third. To "facilitate systematic curricular revisions" and 
"facilitate district-wide assessment of learners and the instructional 
program" were ranked in fourth and fifth position. "Facilitate daily 
testing by classroom teachers" was ranked sixth. 

In response to the question, "What do you perceive as the major 
strength of the COMBAT system?" the majority of the teachers responded 
that it saved a good deal of teacher time. The second most frequent 
response related to "testing for specific objectives" while the third 
most frequent response centered on the advantages of sharing testing 
ideas and objectives with other teachers. In response to the question, 
"What do you perceive as a major weakness of the COMBAT system?" the 
overwhelming majority of responses indicated that the lack of complete 
development of the system reduced its effectiveness for use by classroom 
teachers. These teachers indicated that before the system can be effec- 
tive, the item pool must contain a wider variety of test questions and 
a large number of questions on each specific content area. 
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When asked "What other services would you like to see made available 
from the COMBAT center?" the teacher users suggested the following 
services: test grading and item analysis for teachers, games, lists of 
textbooks and audio-visual materials related to various fields, names of 
people and groups willing to provide special services to the schools 
that allow student exposure to more community "reality experience," 
spelling and vocabulary grade level tests, diagnostic reading tests, 
more ditto masters with labeled drawings to be used as student handout 
study aids, suggestions for planning units to meet specific objectives, 
and m.ore questions on each subject area in specialized programs. 

School district testing representatives were asked, "For future 
development oi canputer-based testing programs, what do you think is the 
greatest potential use for objectives and test items stored in this 
system?" In response to the question, two individuals reported "It is 
a pooling of vast resources, the ability to utili2e and produce in 
v^irlous kinds of areas and knowledges, and a capacity to stor^; and retrl^^ 
information at any particular time by any particular person." Another 
reported "Knowing where your students are in relation to the curriculum 
concepts and what they were taught." Other comments identified the 
greatest potential as "improving individualized instruction" and "for 
producing better prepared and better sducated students, along with the 
possibility to have school building and district standardization." 

I^en asked what additional services they would like to have available 
from this system, the CO>BAT Advisory Board reported the following: 
diagnostic information on the students who have used the syster* , enough 
different grade levels and items so that students can be administered 
tests that are good indicators of achievement, complete item banks in 
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all areas with norms that check for validity and reliability, activities 
that indicate how objectives are to be achieved, modular units (10-15 
items) that can be retrieved rapidly for measurement of selected objec- 
tives and that appear in hard cover. When asked what changes they would 
make in the operating procedures of the system so that it might achieve 
the potentials they described, the members of the Advisory Board reported 
the following: "stress a complete workable file in only one ar^^a where 
teachers can rely on the system and it will run smooothly from the very 
beginning," "more adequate financial support," "start working with indi- 
vidual teachers before moving to a district level," "have a preconceived 
taxonomy where materials can be collected to fill the gaps In specified 
areas," "build in a random access to the computer system," and "make the 
system better known to teachers or at least find some way to get them to 
use the system, and involve the curriculum people more fully in the pro- 
ject operations." 

In order to get a general overall picture of the response of the 
Advisory Board to the total project, they were asked the question, "Do 
you think the computer-based testing service is a sound and practical 
concept?" Ail members of the Advisory Board responded with a definite 
"yes." The testing representatives were asked to rate the total project 
on the following criteria: usefulness, importance, significance, effec- 
tiveness, and feasibility. On "usefulness," eight of the representatives 
rated the project as high, one rated it as very high, two rated it 
average, two rated it low, and one failed to respond. On the "importance" 
criteria, eight representatives rated the project as high, one as very 
high, two as average, and three failed to respond. On "significance," 
nine representatives rated the project as high, one rated it as average, 
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one rated it low, and three failed to respond. On "effectiveness," one 
representative rated the project high, three rated it average, six rated 
the project as low, two rated it very low, and two failed to respond. 
On "feasibility," two representatives rated the project very high, seven 
rated it high, five rated it average, and two failed to respond. When 
asked to c^^plain the basis for their rating on each of the criteria, most 
representatives indicated they rated the "effectiveness" of the project 
lower due to the lack of teacher use of the system. However, the majority 
of the representatives explained their high ratings on the other criterion 
on the basis of the potential for the concept, especially when it is more 
fully developed. The representatives implied that such a system is a 
forerunner of what is to come in education and what will be very useful 
as a teacher time-saving device and as a device for improvement of 
general instructional procedures. 

The interviews with developmental and computer center staff indicac<'I 
that the majority felt the COMBAT project had demonstrated the concept 
to be feasible, but operational and developmental procedures would have 
to be refined before the project could be a truly effective instrument 
for use by classroom teachers. Wien asked to rate the efficiency of the 
COMBAT retrieval system, three of the four members of the computer staff 
felt the project had good potentials for being efficient if more teachers 
would use the system. But in its present state, the retrieval system 
was less efficient than what they would like it to be. A fourth member 
of the computer staff indicated the system was good but it was extremely 
expensive and probably "not worth the money it cost to, produce teacher 
requests." This latter counnent was qualified by a "yes" response to the 
question, "Do you think the effort would be worth the money if more 



teachers would use the program?" Interviews with the key menibers of 
the developmental staff revealed that these personnel felt the sys^jem 
had demonstrated itself to be reasible as a concept, but had not been 
successful in demonstrating the computer-based testing system to be an 
efficient and effective tool for classroom teachers. These staff 
members also suggested that before the system could be an efficient 
and effective classroom tool there would have to be extensive develop- 
ment and refinement of the item files. One ftaff member estimated 
there would have to be at least 100,000 items in order to make the 
total system operational. Another staff member said the project would 
have to spend at least two full years with total concentration on the 
development and refinement of the item pool before it should be open 
for extensive use by classroom teachers. These staff persomiel also 
suggested the emphasis should be on the development of test items rather 
than objectives and items because teachers do not currently think and 
function in terms of behavioral objectives. However, because of the 
potential for teacher use of the behavioral objectives in planning of 
instruction, tht»y also felt th^ development of the behavioral objectives 
should not be totally discarded. In addition to providing a service for 
classroom teachers, the developmental staff identified other potentials 
for the project as its ability to support "expanded indiv? jalized 
instructional programs" and "expanded research into the importance of 
'ndividual student characteristics in an instructional or testing setting,' 

Overall, the majority of the professional personnel exposed to the 
COMBAT system wera positive in their reactions to the efforts of the 
past three years. Although the project was not successful in its f.. ^orts 
to attain wide development and wide use of the system in grade Itvels 



four through twelve, in four subject areas. It has demonstrated that 
computer-based ting is a feasible and potentially useful i.nstructional 
and research tool. In order to make the total sySiiem more fully opera- 
tional and more able to meet its potentials, the greatest need is tor 
a concentrat3d development and refinement effort to complete and upgrade 
the item files . 

E. PERCEIVED EFFICIENCY OF THE PROJECT MANAGEMENT 

The basic management structure of the COMBAT project was somewhat 
involved. The project was funded through Title III of the Elementary 
and Secondary Education Act. Prior to the initial funding, a number of 
agencies were involved developing the plans and proposal for the COMBAT 
project. The team members of the planning effort were staff nembers of 
Teaching Research in the Portland Public School District and school 
districts surroundinp, the metropolitan area, as well as members of che 
County I.E.D. offices from Multnomah, Clackamas, and Washington Counties. 
Thir* Metropolitan Area Testing Program Board (MATPP) was an agency 
formed by the school districts in the three-counuy area. Because t^e 
COMBAT project was related to testing problems and ideas, it was decided 
to make the COMBAT project a part of the activities? sponsored by the 
MATPB organization. The Multnomah County Intermediate Education "district 
wa£5 designated as the fiscal agent for the COMBAT project. At th<i 
insistence of the governmental Title III agencies, a member of the staff 
from the Multnomah County I.E.D. was named project director. The major 
developmental work for the COMBAT project was subcontracted by Multnomah 
County I.E.D. to two other agencies. Teaching Research, a Divi?ion of 
the Oregon State System of Higher Education, received the subcontract 
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to develop the test items and objectives to be stored in the item files 
and to effect the overall coordination of the technical aspects of the 
project) and to provide liaison with the Metropolitan Area Testing 
'Board and the other agencies working with the COMBAT project* Portland 
Public Schools* Computer Center received a subcontract to develop the 
computer storage and retrieval system and to operate the COMBAT center 
for receiving and processing teacher requests. 

This section of the evaluation report is centered on t he efficiency 
of the project management as perceived by the participating a^ncies 
(the project director, representing Multnomah County I,E,D.; the person- 
nel at the Portland Public Schools* Computer Center; the project person- 
nel at Teaching Research; members of the COMBAT Advisory Board selected 
by the MATPB organization; and school district testing representatives 
to the MATPB group). 

The personnel from the different cooperating agencies were asked 
how they were kept informed about the operatiorxS of the COMBAT project. 
The members of the COMBAT Advisory Board stated they were kept Informed 
through regular Advisory Board meetings and meetings of the total MATPB 
organization. A representative of the developmental staff from Teaching 
Research was always present and presented a formalized report. All the 
representatives *:o MATPB, in responding to a questionnaire, indicated 
they received their information from reports by the director of the 
development activities during the meetings of the MATPB oiganization and 
through written communications from MATPB. All members of the Advisory 
Board and the respondents to the testing representatives* questionnaire 
indicated the needed information was always available for the asking. 
Two members of the Advisory Beard pointed out some special communication 



ERIC 



- 79 - 



problems. One member of the Board said there needed to be better communi- 
cation between the general MATPB organization and the COMBAT Advisory 
Board in relation to the overall needs of the MATPB group • A second 
member of the Board stated that although information regarding COMBAT 
was always available, the dissemination of information was not always 
effective because the testing representatives were not always present 
at the meetings. At times there were no more than 25 of the 70 repre- 
sentatives in attendance. He further commented that although the 
needed information was always available, the testing representatives 
seldom took the initiative to inquire about special concerns. 

When the members of the Advisory Board were asked, "Was the general 
management strategy for the project clearly known by yourself and others 
who were involved with the project?" The majority of the Board responded 
"yes." One member of the Board reported "no management strategy was 
displayed." He indicated the "decision making responsibilities should 
have been more clearly defined." All members of the Board commented 
that slightly different strategies were employed by the county I.E.D. 
offices. Teaching Research, the MATPB organization, and individual school 
districts. They further identified the COMBAT Advisory Board as basically 
responsible for decision making regarding project operations, the Mult- 
nomah County Intermediate School District as the final authority for 
fiscal control, and Teaching Research development staff as initiators for 
planning and development of project activities. One member of the 
Advioory Board commented; "there ap-^ared to be some miounderstanding 
about the project responsibilities of the development staff ard the 
responsibilities of the Multnomah County I.E.D." One member of the Board 
indicated that he was unconcerned about the general management of the 
project. 



When asked to describe their role as members of the Advisory Board, 
six of eight individuals reported they were a communication link between 
the teachers. Teaching Research, and the MATFB organization; while three 
individuals indicated th^ provided advice to the COMBAT staff and 
supported the activities of the staff* 

When asked whether or not they were involved in project decision 
making, the majority of the members of the COMBAT Advisory Beard reported 
that they were involved with all of the basic decisions regarding the 
COMBAT project operation. All of these Board members reported the pro- 
ject director and the director of the uevelopment staff were very agree- 
able as far as accepting the ideas and decisions of the Board and in 
taking steps to implement the ideas* Of the 14 district representatives 
responding to a questionnaire, nine indicated they were not directly 
involved in project decisions, four indicated they were involved, and 
one failed to respond* Three of the four ''y^^*' responses were qualified 
as follows: "somewhat, participated in the original planning grant," 
"All policy decisions," and "Secondary, not in project development." 

All members of the project staff at the Computer Center reported a 
lack of communication between the Computer Center and the MATFB organiza-* 
tion* Three of the four staff members interviewed indicated they did 
not know what the MATFB organization was all about, or how the COMBAT 
project was associated with the organization* Although members of the 
computer staff found the general quality of communications between 
themselves and the developmental staff to be reasonable, they felt they 
should have been kept better informed about the relationships of the 
total project to the MATFB program, and felt that communications between 
themselves and the developmental staff could have been better. Two of 
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the primary reasons for some deficiencies in communications between the 
computer staff and the developmental staff were listed as the distance 
in mileage between the two organizations and the fact that few of the 
developmental staff had a thorough knowledge of the computer capabilitl.- • 
and operations. The members of the developmental staff said the cjimnuni- 
cations between themselves and the MATPB organization were very good, 
but they also recognized the existence of some communications problems 
between themselves and the computer staff. 

Members of the Advisory Board and the two technical staffs of the 
COMBAT project were asked to identify the nost critical project decisions 
as they perceived them. The following responses were recorded from the 
COMBAT Advisory Board. ''The decision to involve testing people instead 
oi curriculum people was a mistake. If curriculum people would have 
been involved, it would have been a more solid fiinding base and a closer 
contact with teachers." Another comment was, "Tixe decision on what 
area the initial item pool would cover and the program for input and 
output of data. The real problem was the COMBAT project was originally 
to be four or five years, but was cut to three years/* A third member 
of the Board stated that, "The time to finally make the project opera- 
tional was a critical decision." Another member reported, "They should 
have stayed with the fourth, fifth, and sixth grade social studies and 
not expanded to other areas"; and "enough money should have been budgeted 
to supply teachers for writing objectives and items." Another member 
of the Board stated, "IWo critical decisions were the expansion of the 
item files and the use of teacher constructed tests as a source for iteniB 
Aixother Board member indicated the critical decision was "the change 
from fourth, fifth, and sixth grade social studies to a more broadly 



defined area." This member felt that this decision was a "bad" in error. 

Members of the developmental staff reported that, in their opinion, 
the most critical project decisions came initially with the choice to 
center the early work of the project with grades four through six in the 
social studies areas. They felt this decision made the initial work in 
the development of the system more difficult than if a higher grade level 
and a more well defined instructional program or curriculum area had been 
selected. Other critical decisions were identified as the move from 
periodic workshops for obtaining test items to the D.C.E. classes in the 
second year, and ilnally to the decision to edit items supplied by 
teachers from the tests teachers wrote. 

The Computer Center personnel identified two major changes in the 
project operation as affecting the project. The first was the switch 
from a small to a large computer at the Portland Public Schools* Computer 
Center. The impact of this change resulted in increasing the cost of 
operating the COMBAT center. The other major change was identified as 
the ^itch in the output formats that were mailed to teachers of the 
result of their initial request. The latter decision was not a major 
problem hoijever. One member of the Computer Center staff indicated 
a problem resulted from the decision to edit teacher constructed tests. 
Since college students were trained to edit and revise these items, the 
work of these individtials should have been checked more closely. This 
member of the staff Indicated that much of the material sent to the 
Computer Center from the student item-editors contained several mistakes, 
such as spelling, grammar, and punctuation. This staff member identified 
the extistence of these poorly constructed items as a crucial weak point 
in the system. Although efforts were made to identify and correct 
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erroneous or poorly constructed test items, the procedures were not 
adequate to completely refine all of the items. 

In response to the question, **Did any special circumstances and/or 
constraints either help or hinder the operations of the COMBAT project?*', 
6 respondents to the district testing representatives' questionnaire 
indicated "yes," two indicated "no," and six failed to respond. When 
asked to list those circumstances which helped the project, two of these 
people indicated that the "chairman was eager, enthusiastic, and built 
enthusiasm," and one representative indicated the "increased utilization 
when teachers were asked to submit their own test items" for the item 
bank. Other comments were "The exchange of ideas with other districts 
needed in getting more needed items, the project director assuming a 
more appropriate role as director not Just fiscal agent , dinners for 
teachers who might use or encourage the use of the system." One repre- 
sentative indicated he had been on the Board only for one year and 
couldn't make a value judgment. When asked to list those circumstance^^ 
which hindered the project management, the followir^ responses were 
recorded: Three representatives identified the lack of funds for teacheiY 
to develop materials, two identified thj reluctance of teachers to accept 
the proiect, two identified poor communications between the districts 
and COMBAT, and "COMBAT's overstating its ability to perform," "the pro- 
gram was not used by the extremely wide geographic area to be covered" 
"the distance between the managraent and this area," "the superintendent 
did not understand and was opposed to it," "index was distributed too 
late," and "the request that the tests used by teachers be ret :med to 
Teaching Research for item analysis discouraged teachers from using the 
system." 



When the members of the COlfBAT Advisory Board were asked, "What 
special circumstances and/or constraints either helped or hindered the 
operation of the CO^IBAT project? the following responses were recorded: 
"One hindrance to the project was the political Interplay between the 
Intermediate Education Districts and the individual school districts » 
but COMBAT had no constraints," "it helped to let the teachers know the 
system was developmental and not operational when they were writing items 
and objectives, because some teachers dldn*t see the need for this 
activity if it was going to be operational at the beginning," "the human 
resources and the organization Involved were the most valuable in helping 
to implement COtfBkT/* "newness of the idea and the limited' curricular 
areas probably hindered the project," "It Is hard to sell an idea when 
teachers aren*t Interested in a service," "cutting of the funds by the 
national funding agency" "Division^ of Management, the I«£«D«, who was 
supposed to be the director but was nothing more than a name on a piece 
of paper for the first year. The Advisory Board did not function in the 
capacity to the extent they should have," "the main hindrance was communi- 
cation between the MATPB organization and the population of teachers 
and administrators given the matter of planning." 

The members of the Computer Center staff Indicated the major hin- 
drance to project operation was the difficulty in obtaining feedback from 
classroom teachers regarding the quality of the items they received from 
the COMBAT center. They Indicated it was difficult to attempt to refine 
uhe items in the system when teachers did not consistently respond to 
requests for their reactions to the items they received. The member of 
the developmental staff identified the iiajor problems as the distance 
between the major subcontracting agencies ana Llie Lime it takes to get 
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from one to the other. They indicated that it was hard to maintain 
conm'inications between these two agencies. However, they indicated that 
communication with the MATPB organization was good because the Advisory 
Board meetings always immediately proceeded the general board meetings y 
and because of this "all of the right people were there at one time." 
Another problem identified by the developmental staff were changes xn 
personnel within the developmental effort. The project would have been 
more efficiently operated if there had been a single major director of 
this activity over the full three years. One staff member of tl*e develop^ 
mental group indicated tliat most of the money from the subcontrac let 
to the Computer Center went for salaries, while most of the computer 
time was donated to the project. This condition lowered the priority 
on computer time that as given to the COMBAT project. 

The questionnaire designed for school district representatives to 
the MATPB organization included two questions that asked tham to rate 
the project management. The first asked each representative to rate th<. 
degree of efficiency of the project i^anagement on five basic scales. 
In response to the first scale, consulting with MATPB representatives ; 
three of fourteen rated efficiency as very low, one rated it as low, 
three rated it as average, four as high, and three failed to respond. 
In response to: use of teachers for system development ; four rated the 
efficiency as low, four as average, three as high, and three failed to 
respond. On the scale, use of computer facilities ; four representatives 
rated it average, five rated it high, and five failed to respond. On 
the seal", use of project staff ; three representatives rated t.ie effi- 
ciency low, five average, two high, and five failed to respond. On the 
scale, use of budgeted financial resources ; four representatives rated 
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the project as average, three as high, and six failed to respond. Two 
of che fourteen representatives volunteered a low or very low rating for 

4 

the project management on public relations with school districts. 

The second rating item asked the district representatives to "gen- 
erally rate the project management on effectiveness and efficiency",. Of 
the fourteen representatives who returned the questionnaire, five rated 
the effectiveness as low, five as average, one as high, and three failed 
to respond. On efficiency, three rated low, five average, two high, and 
four failed to respond. In general, these reponses indicate the district 
representatives were not overly impressed with the general management of 
the project. The representatives seemed to be particularly conceraed 
with the ability of the project staff to communicate and interact with 
the teachers and other professional staff in individual school districts. 

The members of the Advisory Board were asked to rate the use of 
project resources, i.e.; the use of the Advisory Board, the use of teachers 
and other school district personnel, staff time, and financial resources. 
Five members of the Advisory Board commented in response to this question. 
Four rated the use of the Board's time as high and one rated it average. 
In response to; the use of teachers and other school district personnel ; 
three members rated the project as high or average, while two Individuals 
rated this item as low. Four of five members of the Advisory Board rated 
the use of staff time as high and one individual rated it as average. 
All but one of the members of the Advisory Board declined to rate the 
project management on the use of financial resources because the>- were 
not involved with the financial aspects of the project. The one member 
of the Advisory Board that did comment tated the use of available 
financial resources as high. Another individual declined to rate the use 
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of financial resources, but indicated that he would like to have re- 
directed the first and second year so that the staff and computer time 
would have been used for the production of high quality materials. 
Members of the Advisory Board were also asked tc generally rate the pro- 
ject management on effectiveness and efficiency. Seven members of th^ 
Board responded to this question^ Four individuals rated the project 
management high and three rated it as average on both effectiveness and 
efficiency. 

Overall, most of the people involved with the COMBAT project tended 
to rate the management of the project in the average range. However, 
there was some fluctuation with the rating depending upon the aspect 
of the management being discussed. Project management was rated lowest 
in its efforts to work with individual school district personnel in 
developing the item pool. The basic problem existed with public rela- 
tions and general communications between researchers and practitioners. 
The literature in educational research shows this type of communication 
problem to be consistently present when researchers and practitioners 
collaborate on a development project. The data indicates that the COMBAl 
project '-^s not able to bridge this gap any better than the majority of 
projects that require the cooperation of researchers and practitioners. 
As far as the technical competencies are concerned, the project manage- 
ment and operations .tended to be rated slightly better than average. 
That is, the technical competencies of the computer staff and the develop- 
ment staff at Teaching Research were not questioned. Part of the prob- 
lem may have been the shifting of staff on the developmental team. The 
shift in project personnel caused some communication problems between 
the developmental staff at Teaching Research and the staff at the Computri 
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Center. It also caused some communications problems between the develop^ 
mental stafi and the MATFB organization. However, this latter problem 
was small. 

The project decision chat received most comment by people in all of 
the agencies involved with the project was to attempt implementation 
before the item pool was fully developed • Several individuals identified 
this decision as leading to teacher dissatisfaction with what they 
obtained from the system. It appears that most teachers expected the 
system to be completely developed before it was disseminated for 
implementation. This misunderstanding led to dissatisfaction and 
frustration in teachers. The majority of personnel involved in the 
project have indicated that to make the project more functional, the 
major emphasis should be upon the development task. That is, the item 
pool needs to be greatly expanded and refined in order to make the 
system more functional for classroom teachers. 



IV. INTERPRETIVE SUMMARY OF FINDINGS 

A. CONGRUENCY BETWEEN OBJECTIFIES AND OUTCOMES 

At the end of a three-year period the COMBAT project was able to 
generate 32,163 test items and objectives. It is not known how tnanv 
different specific instructional objectives were included in this list^ 
For each test item In the item pool there was a corresponding instruc- 
tional objective. However, some objectives had more than one test item 
associated with them* Although a large number of items were identified, 
the project was not able to identify instructional areas for all ele-* 
mentary and secondary curricular areas at all grade levels • Although 
many items and objectives were identified for grade levels four through 
twelve in Social Studies, Language Arts, Mathematics, and Science, 
teachers were not satisfied with the breadth and depth of the item pool 

Although the Computer Center was successful in designing a storage 
and retrieval system that does function, there is a high cost for pre-- 
duction of teacher requests of list of test items and test dittos. The 
telephone*-mail system designed to serve as a communication link between 
t:he teachers and the COMBAT center appears to function well and to be 
well received by teachers* In the opinion of the personnel at the 
Computer Center, it would be difficult for teachers, with access to 
terminals, to use the present system for retrieving test items directly 
because of the knowledge of computer language required to search the 
files. This latter reason is also justification fo®the telephone 
communication system. 
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The findings of this evaluation indicate that inrservice programs 
to train teachers in use of the (JOMBAT system were insufficient for the 
needs of the large service area included in the target population of 
the project. There is little direct evidence that the COMBAT project had 
any impact on the overall effectiveness of instruction m the curricular 
areas for which objectives and test items were prepared* Although the 
majority of teachers who responded to a questionnaire indicated that ' 
the COMBAT system made no change in their instructional programs, a 
small percentage indicated that it had some impact on the frequency of 
testing and the degree to which they planned instruction f. > specific 
objectives . 

Although numerous methods for disseminating information to the 
teachers in the service area were attempted, there are large numbers of 
teachers in the service area who were not informed about the COMBAT 
project* Dissemination of information to classroom teactiers is con- 
sistently a problem with innovations in education* This is especially 
true when the innovation is initiated from outside of the school district 
and the administrative personnel within the district are not directly 
involved in the project. Procedures to overcome the problems assoc- 
iated W3th intormation dissemijiation under such circumstances were 
not completely overcome by the COMBAT project. 

During the third year of the project the staff were successful 
in expanding the item files by over twenty thousand objectives. They 
also reached their objective to e>rpand the number of schools who had 
one or more teachers using tne COMB/iT system from 79 to 150 elementary 
schools and to introduce the CX)MBaT system to at least half of the 
secondary schools. The records indicate that the staff was able to 
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get at least one teacher in almost every secondary school in the service 
area to attempt to use the COMBAT system at least once. However, the 
number of teachers that used the system was limite^. In all, a tot^^l 
of 470 teachers were recorded as having used the system by raid-53Dring. 
1970. The majority of these teachers made only one or two renvr^t^ 
seldom if ever made a second request for the production of a test dxttc 

The developmental staff were successful in producing an index of 
key words for distribution to teachers In the four content areas at 
the different grade levels. However > of the teachers who had used the 
system, less than 20 percent reported they had seen a copy of this index. 
There was an apparent breakdown in the procedures for distributing these 
documents from the develoimental staff through the county I.E.D. offices 
to the school district offices and eventually to individual school 
buildings and teachers. 

B. CONTINGENCY RELATIONSHIPS WITHIN PROJECT 
OPERATIONS AND OUTCOMES 

The most critical relationship between the project operation and 
outcome was identified as the degree of development that is needed 
before teachers will use the system. There, is evidence to support the 
notion that the majority of teachers were thinking of the COMBAT system 
as completely developed rather than an emerging testing system. When 
teachers attempted to use the system and found it was not totally opera- 
tions! at all levels they were discouraged from further use. Part of 
this discouragement can be attributed to what some members of the MATPB 
organization called "overselling" of the project to the teache. '?. It 
is strongly recommended that for future development of projects similar 
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to COMBAT that development be more fully completed before an attempt is 
made to obtain wide usage by classroom teachers* 

There is also some evidence that operation of the project could 
have been improved if project personnel had not been changed. Th<x- is, 
if there had not been as many changes in the personnel assigned to the 
project in the developmental center at Teachin]^ Research. A yearly 
change in project directors for the developmental effort caused some 
communication problems with the computer staff and with the general 
continuity of the total project. 

C. SPECIAL PROBLEMS RELATED TO PROJECT CONTEXT 

The large size of the area serviced by the 03MB AT project presented 
some special problems to the project. These problems include the extremely 
diversified instructional objectives and/or strategies used by the many 
different school districts and a large number of teachers tor the spec- 
ified curricular areas and grade levels to which the project was directed. 
With the time and mon^ resources » it was not possible to adequately 
identify all of the instructional objectives and develop appropriate test 
items* Another special problem related to the size of the service area 
was the physical capacity for the project staff to interact with key 
people in each of the seventy school districts and over three hundred 
school buildings. This circumstance led to special problems in dissem- 
ination of information and in the training of key personnel in each 
district. The COMBAT project was a function of the MATPB organization, 
but there is an indication that few districts identified the COMBAT pro- 
ject as a part of the MATPB program. Although many districts heavily 
supported the project, there is an indication that several school 
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districts did not understand the purpose or nature of the COMBAT pro- 
ject and therefore did not identify with it. 



id 
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V* GENERAI. CONCLUSIONS 



In general it can be concluded that the COMBAT project was moder- 
ately successful. The project demonstrated that a computer-based 
testing system can be built, but failed to demonstrate it as an effec- 
tive and useful tool by classroom teachers for improving classroom 
instruction. 

The demonstration of the technical feasibility of a con^uter- 
based testing program is a significant contribution to the field of 
education. Howevev, with the present storage and retrieval system 
the cost of retrieving lists of items and the cost of producing test 
dittos for teachera remains markedly high* These costs can be reduced 
with the present system by increasing the number of teachers who make 
use of the system* In spite of this potential for reducing the cost 
for retrieving items, it is strongly recommended that the Computer Center 
investigate alternative strategies for retrieving materials stored in 
the item pool. In general, teachers are satisfied with the telephone- 
mail communication system between themselves and the Computer Center. 
This system appears to function well. 

Teachers tend to be satisfied with the test items and objectives 
that are stored in the system, but indicated a dissatisfaction with the 
diversity and coverage that were developed in the three-year period. 
This dissatisfaction could have been overcome if the project wou7d have 
lindted'its efforts to a smaller curriculum area. The project would 
probably have been able to more adequately demonstrate the utility of 
the system for classroom teachers and its impact upon instruction if 
such an alternative procedure had been followed. The decision to e3q>and 



the grade levels from four through six to four through twelve » and 
from one currlcular area to four currlcular areas ^ spread the develop- 
mental staff too thin. Not enough materials could be developed in each 
of these areas to make the system satisfactory to enough teachers. 

Before the present system can demonstrate its effectiveness as an 
instructional tool and an aid to classroom teachers » the item pool will 
have to be greatly expanded. This means that the item pool will have 
to be enlarged by three to four times its present rumber. In addition , 
the system should be refined by working through small groups of selected 
teachers rather than through the general teacher population in a given 
geographic area. On the whole» teachers expect projects to be fully 
developed before they are asked to become involved except under very 
special circumstances. Teachers that are involved in the development 
and refinement efforts will probably have to be financially reimbursed 
for their work. 

There were some communication problems within the various agencies 
included in the OniBAT project* TWo major communication problems existed. 
One of these was the dissemination of information from the project staff 
to teachers at each school buildli^. The task was too large to be 
adequately handled by the project staff itself » and the delegation of 
communication responsibilities through the hierarchy of educational 
institutions did not operate effectively. Second^ there were some 
communication problems between the developmental staff and the staff 
at the computer center. Part of these problems resulted from the staff 
turnover within the developmental group and because the developmental 
staff did not always have a working knowledge of computer systems. 
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Although It mi^t not be entirely necessary for the developmental staff 
to have a knowledge of the computer systems ^ the coiiq)uter staff in the 
present project indicated that this condition caused some problems. in 
communicat ion • 

The management of the technical aspects of the project were gener-* 
ally rated as better than average. However^ because of the inability 
of the project staff to get a large proportion of teachers to consistently 
use this system, the management tended to be rated somewhat lower on 
effectiveness* From a post hoc position, members of the MATPB COMBAI 
Advisory Board and school district representatives to the MATPB organ- 
ization questioned the project decision to expand the project target 
population. Some members of the MATPB organization questioned the 
effectiveness of the developmental staff in working with school district 
personnel. This latter comment is not unique to the OOMBAT project. 
The literature of educational research consistently reports the problems 
of conraunication that exist when researched and practitioners cooperate 
on developmental projects. 

Personnel from all of the agencies that cooperated on the COMBAT 
project were consistent in their agreement with the potential power for 
the concept being explored by the C<»fBAT project. These professional 
educators all agreed that once the concept is more fully developed and 
refined that it will make a significant contribution to Education. 
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APPENDICES 



Appendix A 
System User Questionnaire 



A Titta til Project 



COMPUTER BASED TEST 

Development 
Center 




COMBAT Evaluation Ouastionnaira 

Form I 



Evaluation Unit (^3 
TEACHING RESEARCH 

41 Dhi$tQn ot th0 Ot^gon Sfafa $f$t9m of Mtgh*r Education 

Monmoulh. Oregon 



COMBAT USERS* QUESTIONNAIRE 



1. 



□ 


1 - 


3 


□ 


4 - 


6 


□ 


7 - 


9 


□ 


9 - 


10 




11 - 


12 


□ 


9 - 


12 



2. 



□ 
□ 
□ 
□ 
□ 

3. 

□ 
□ 
□ 
□ 
□ 

4. 

□ 
□ 



What is the grade level of the majority of the students 
you teach? 



Check the box which best describes your major 
teaching assignment? 

Self-contained 

Science 

Mathematics 

Social Studies 

Language Arts (not a foreign language) 
Other (please specify: 



What is your age? 

20-30 

31 - 40 

41 - 50 

51 - 60 

61 - over 

Sex 

Male 

Female 



Which of the following best describes the grade levels 
included in your school? 



1 - 6 
6-8 
7 - 9 
10 - 12 
9-12 



Which of the following best describes your present level 
of college training in terms of degree and term (quarter 
hours? 

Bachelor degree 

Bachelor + 20 Hrs. 

Master's or Bachelor + 45 Hrs. 

Master's + 20 Hrs. 

Master's +-45 or more Hrs. 

How many times have you requested a list of test iten>s 
from the COMBAT system? 

0 

1-2 
3-4 
5-6 
7 or more 

Approximately what percent of your total classroom tesi.: 
could be done with test items obtained from the COMBAT a 



0 - 20% 



21 



40% 



41 



60% 



61 



80% 



81 



100% 



Diificulty 



Range of Content 



Range of Item Style 



Clarity 



10. 



□ 

□ 
□ 

□ 

□ 

□ 



11. 



Quality of tests 



Indicate your general rating of the list(s) of test 
items you have received on each of the following 
criteria. 



LJ 


n 

1 1 


n 
1 1 


n 
1 1 




□ 


Very 


Hard 


About 


Easy 


Very 


Hard 




right 


□ 


Easy 


□ 




□ 




□ 


Very 


Similar 


Ah out 


Diverse 


Very 


Similar 




right 




D: 


verse 


□ 


□ 


n 








Very 


Simlls 




Diverse 


Very 


Similar 




.igftt 


□ 


Jiverse 


□ 


□ 


□ 






Very 


Ambiguous 


Clear 


Very 






Ambiguous 






Clear 







If you have ever requested both objectives and associated 
t:^<it items > RATE the appropriateness of the test items for 
t'le associated objectives. 

Very low 

Low 

About 50% appropriate 
High 

VeiTf High 

Never requested both 



How did the COMBAT system change your classroom 
instruction in each of the following areas: 

□ □ □ □ □ 

Much Worse No Better Much 

Worse Change Better 



11. 


(Continued) How did the COMBAT system change your 
classroom instruction in each of the following areas: 




Much 
Less Often 


Less 
Often 


No 
Charge 


More 
Often 


Much 
More Often 


Frequency of testing 


□ 


□ 


□ 


□ 


□ 


, cy of pre-testlng 


1 — 1 


1 — 1 
1 — 1 


1 — 1 
1 1 


1 1 


1 — 1 


Plan tests related to 
sneclflc ohlectives 


□ 


□ 


CZl 


n 


□ 


Plan instruction for 
specific objectives 


□ 


□ 


□ 


□ 




Used tests in new ways 


□ 


□ 


□ 


□ 


□ 


Used objectives in new 
ways 


□ 


□ 




□ 


LJ 


Time required for test 
construction 


□ 


□ 


□ 


□ 


□ 




Much 
More Time 


More 
Time 


No 
Change 


Less 
Time 


Much 
Less Time 


Size of group tested 


□ 


□ 


□ 


□ 


□ 




Much 
Larger Group 


Larger 
Groups 


No 
Change 


Smaller 
Groups 


Much 
Smalle:. Groups 



12. 

□ 
□ 

□ 

□ 

□ 
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Which of the following best describes what you did with 
the list of test items received from the COMBAT center? 



Did not use the test items wJth students 

Selected a few items and added several of your own 
before preparing a test ditto 

Selected several items and added several of your 
own before preparing a test ditto 

Selected items only from the list and prepared your 
own test ditto 

Selected items only from the list and ordered a test ditto 



13. 



□ 


0 


□ 


1-2 


□ 


3-4 


□ 


5-6 


□ 


7 or more 



14. 

□ 
□ 
□ 
□ 
□ 

15. 

□ 
□ 

16. 

□ 
□ 
□ 
□ 



How many times have you requested a list of objectives 
from the COMBAT system? 



RATE the appropriateness of objectives requested for your 
intended purpose. 

Very low 

Low 

About 50% appropriate 
High 

Very hl^ 



Has the COMBAT center always sent back the test items 
you selected for the final test? 

Yes 

No 



How long did it take (in elapsed school days) for you 
to receive the list of test items. 

1-2 days 

3-4 days 

5-6 days 

7-8 days 




17. 

□ 
□ 
□ 
□ 
□ 

18. 



Readable 

Language Understandable 

Form of Question that 
you approve of using 

Ease of Scoring Test 

Clarity of Symbols 
used (as appropriate) 

19. 



Ease of reading 

Economical use of 
space on page 

Dittos make adequate 
number of copies 

Dittos are coiqpatible 
with machine at ny 
school 



liow many times have you requested a final test (ditto form) 
from the COKBAT system? 

0 



1-2 



3-4 
5-6 



7 or more 



Please rate the formal or appearance of the final test 
according to the following criteria. 

Very High Moderate Low Very 

hi^ low 

□ □ □ □ □ 



□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 



Please rate the test dittos you received on the following 
criteria: 

Very High Moderate Low Very 

hlgfi low 

□ □ □ □ □ 

□ □ □ □ □ 

□ □ □ □ □ 



20. What changes In the print-out format for objectives and 
test items could you suggest that would make it (the 
print-out) more acceptable? 



a. 
b. 
c. 
d. 



21. Please make one positive comment and one negative comment 
about the procedures for obtaining objectives, lists of 
test items and test matters. 

Positive comment: 



Negative comment: 



22, Do you have access to the COMBAT COMPUTER-BASED TEST 
DEVELOPMENT INDEX for your grade level and subject area? 

□ Yes 
□ 

23. If you answered yes on number 22 , rate the index on the 
following dimensions of usefulness as an aid in requesting 
lists of objectives and/or test items from the COMBAT 
sys tern. 



Usefulness : 


□ 


□ 


□ 


□ 


□ 


□ 




aot of 

useful 


little 
use 


useful 


very 
useful 


essential 


don' t 
know 


Complexity: 


□ 


□ 


□ 


□ 


□ 


□ 




too complex 
to be used 


slightly 
complex 


coQq)lex 


not too 
coiq>lex 


very 
clearly 


don* t 
know 


CoD^rehens iveness : 


□ 


□ 




□ 


□ 


□ 



too slightly con5)rehensive just don't 

comp rehens i ve too r i gh t know 

conq)rehensive 



erJc 



24. 

□ 
□ 
□ 
□ 
□ 

n 
□ 
□ 

25. 

□ 
□ 
□ 
□ 
□ 

26. 

P 
□ 

□ 

□ 

27. 

□ 
□ 
□ 

n 



Place a check In the box corresponding to each of the 
following ItemB you have seen. 

COMMT BROCHURE 

Telephone stickers advertising C(»JBAT 
COMBAT slide-tape presentation 

COMBAT article in the journal OREGON EDUCATION, December, 1970 
COMBAT presentation at O.E.A. State Convention, 1969 
Matchbooks advertising COMBAT 
COMBAT puzzles 
none of these 

How do you rate your familiarity of the COMBAT SYSTEM? 

near zero 

low 

barely enough to use it 

moderate 

high 

What was the approximate date of your LAST request 
from COMBAT? 

Sept. - Dec, 1968 

Jan. - June, 1969 

Sept. - Dec, 1969 

Jan. - May, 1970 

If you have ordered one or more lists o€ objectives, 
which of the following best matches the reason(s) for 
your reqtxest? 

To e:q>eriment with the COMBAT system 

To use the objectives as a guide for planning instruction 
To search for new ideas for classroom instruction 
To use the objectives as a guide for ordering test: items 
Other (specify) 



28. 



A pretest 

A post test 

A diagnostic test 

A special test 
designed for one. 
studen.t and a 
smaJ 1 group of 
students 



Pretest 

Post test 

Diagnostic test 

To dieck on 
instructional 
effectiveness 
but not to 
grade students 



29. 



30. 

□ 
□ 

□ 
□ 
□ 



Items for: 



0 12 


3 


4 or moxkt 




n n n 

LJ 1 1 1— ' 


n 

1 — I 


n 




□ n □ 


□ 


□ 




n n n 


n 

1 1 


□ 




□ □ □ 


□ 


□ 




Approximately how often do 


you test, in 


your own instructional 


program, for each of the following purposes? 




Never 1-3 


4-6 


1-2 


3 or xnore 


times/yr. 


times /yr. 


times/mo. 


times/roo. 


□ □ 


□ 


□ 


□ 


□ □ 


□ 


□ 


□ 


□ □ 


□ 




□ 


□ □ 


□ 


□ 


□ 



What would be the Ideal number of school days between the day 
you order a list of test items and the day you receive 
the list? 

0 

1 

2 

3 

4 



What would be the maximum number of school days that can 
elapse between the day you order a list of test items and 
the day you receive the list? 



0-1 
2-3 
4-5 
6-7 
8 or more 



Approximately what percent of your student grading or 
evaluation is based on tests coiq)Osed of items like 
those stored in the COMBAT system? 

0 - 20% 

21 - 40% 

41 - 60% 

61 - 80% 

81 - 100% 



Please rank the following possible uses for the COMBAT 
system in order of iiqportance. The most important should 
be rated one (1) and the least important six (6). 

Facilitate daily testing by classroom teachers. 

Facilitate systematic curricular revisions. 

Provide feedback for students on their current level of 
learning (achievement). 

Facilitate district wide assessment of learners and the 
educational program. 

EiUiance the teacher's role as ^'instructional manager." 

Test the effectiveness of specific instructional methods 
or materials. 
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34* Place a check In the box corresponding to each of the 
following activities In irtiich you participated. 

\ I COMBAT teacher workshop - Sunmer, 1968 (Writing obj., etc.) 

I I COMBAT Initial e^qperlmental trials - Fall, 1968 

rn Department of Continuing Education workshop measuring 
Educational Objectives - Fall & Winter, 1968-69 

I I Building or district level CCXIBAT orientation meeting - 
Fall & Winter, 1969-70 

{ZD COMBAT teacher advisory team dinner - October, 1969. 

35. What do you perceive as t^se m^jor strength of the C(»1BAT 
system? 



36. 



What do you perceive as the major weakness of the COMBAT system? 



37. What other services would you like to see made available 
from the COMBAT center? 



a. 
b. 
c. 
d. 



Appendix B 
General Questionnaire 



A Title ill Profect 



COMPUTER BASED TEST 

Devolopment 
Center 




COMBAT Evaluation QuMUonnaire 

Form II 



Evaluation Unit 



TEACHING RESEARCH 

• DM$lon of thm Omgo/t Sfefe Syi fern of HigH§f education 

Monmouth. Oregon 



COMBAT 
General Questionnaire 



1 1 

i 1 


Uhaf iti thp i23ra.de IpvpI of the maiorltv of the students 
you teach? 

1-3 


^11 


4-6 


□ 


7-9 




9-10 


□ 


11 - 12 


u 


9-12 


2 

1 


Check thp box associated with the best description of 
your major teaching assignment* 


□ 


Sel f-contalned 


1 — 1 
1 — 1 


o v> xcu w>c 


1 — 1 


OvW>XaX *^ u-uuxca 


1 — 1 
1 1 


Ma thema. t i. c s 


1 — 1 
1 — 1 


LAnciiAce Arts (not foreien lan£ua£e) 


□ 


Other (Specify) 


3. 


What is your age? 


n 


20-30 


□ 


31-40 


□ 


1 « 

41-50 


u 
1 1 


51-60 
61 - over 



4. 


Sex: 


n 


Male 


□ 


Female 


5. 


Which of the following best describes the grade levels 
included in your school? 


1 1 

□ 


Bachelor degree 


n 


Bachelor + 20 hours 


□ 


Master's or Bachelor + 45 hours 


□ 


Master's + 20 hours 


□ 


Master's + 45 hours or more 


6. 


Which of the. following best describes the grade levels 
included in your school? 


1 1 

LJ 


1-6 


□ 


6-8 


□ 


7-9 


□ 


10 - 12 


□ 


9-12 


7. 


How often do you test in your classroom? 
(Check each appropriate caf^gory.) 


a 


After each individual instructional unit 


□ 


After each MAJOR instructional unit 




Daily 


□ 


Weekly 




Monthly 


□ 


Quarterly 


□ 


As it seems appropriate 


□ 


Each grading period 
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Why do you give tests to your students: 
(Check each appropriate category.) 

To plan Instruction 

For grading and reporting student progress 
To monitor student progress 
To diagnose student weaknesses 

To provide guidance for students' Indlvldtial study 
To motivate study (both group and Individual) 

a. Do you mainly obtain test Items from the Information 
contained in your Instructional materials, such as 
t xt books, films, recordings, etc.? 

jfes 
Wo 

b- If YES, do you use Items: 
Prepared by the textbook publisher? 
Write your own Items? 

a. Do you use any special references In constructing 
tests, such as lists of test Items? 

Yes 
No 

b. If YES, please name one such reference. 



Place a check In the box corresponding to each of the 
following Items you have seen. 

COMBAT Brochure 

Telephone stickers advertising COMBAT 
COMBAT slide-tape presentatlor. 

COHBAT article in the journal OREGON EDUCATION, December, 1970 



11. 

□ 
□ 

□ 

□ 

12. 



o 
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Pretest 

Post- test 

Diagnostic test 

To check on instructional 
effectiveness but not to 
grade students 



13. 

□ 

□ 
□ 
□ 

14. 

n 
□ 
□ 
□ 

CD 



(Continued) Place a check In the box corresponding to 
eac' of the following items you have seen. 

COMBAT presentation at O.E.O. State Convention, 1969 

Matchbooks advertising COMBAT 

Copies of COMBAT test items or test ditto masters 
None of these 



Approximately how often do you test for each of the 
following purposes? 



1-3 

Never times/yr. 



□ 
□ 
□ 



□ 



□ 



□ □ 



4-6 
times/yr. 

□ 
□ 
□ 

□ 



1-2 
times/mo. 

□ 

□ 

□ 

□ 



3 or more 
times/mo « 

□ 
□ 
□ 

□ 



In order to make a testing service most effective what 
would be the ideal number of school days between the day 
you order a list of test items and the day you receive 
the list? (For any test purpose.) 

0 

1 



2 
3 
4 



Vlhat would be the maximum number of school days that can 
elapse between the day you order a list of test items 
and the day you receive the list? 

0-1 

2-3 

4-5 

6 - 7 

8 or more 



15. 



□ 
□ 
□ 



□ 
□ 
□ 

16. 

□ 
□ 
□ 
□ 



Where Is the nearest pho{ie, at school, that you can use to 
call the COMBAT center? 

Your office 

Department office 

Teacher's lounge 

School secretary's office 

Principal's office 

Public pay phone 

Other (Specify) 



How easily available Is this phone? 
Easily available 
Fairly convenient 
Moderately inconvenient 
Available but not convenient 
Unavailable 



17. If you intended at this moment to call the COMBAT system 
for any reason, how easily could you locate the telephone 
number? 

I I I know (xactly where the number is 
I I I think I know where I wrote it down 
I I I know who should have it 
I I I know who might have it 

I haven't the slightest idea who might have it 

18. Place a check in the box corresponding to each of the 
following activities in which you participated. 

I I COMBAT teacher workshop - Summer, 1968 

I I COMBAT initial experimental trials - Fall, 1968 

I I Department of Continuing Education Workshop Measuring 
Education Objectives - Fall & Winter, 1968-69 
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(Continued) Place a check In the box corresponding 

to each of the following activities in which you participated. 

Building or district level COMBAT orientation meeting - 
Fall & Winter, 1969-70 

COMBAT teacher advisory team dinner - October, 1969 

a. Have you ever requested test items from the COMBAT 
system? 

Yes 
No 

b. If YES, approximately how many times? 
1-2 

3-4 
5-6 
7-8 
9 or more 

a. Have you ever used test items from the COMBAT system? 
Yes 

No 

b. If YES, approximately how many times? 
1-2 

3-4 
5-6 
7-8 
9 or more 



Appendix C 
Metropolitan Area Testing Prograa Board 
Representative Questionnaire 



A Title lit Project 



COMPUTER BASED TEST 



Development 
Center 



COMBAT Evaluation QUMtionnaire 

Form III 



Evaluation Unit Q 
TEACHING RESEARCH 

e Di¥i»ioft of Me Oregon Sfefe SyMfm of Hlghor Sducttion 

Monmouth* Oregon 



COMBAT - MATPB questionnaire 



Name: Date: 

School (school district): 



Briefly describe your present district position (job):_ 



Job title: Yrs. In this job:_ 



Regarding The COMBAT System... 

1. For future development of computer based testing programs, 
what do you think Is the greatest potential use for: 

(a) objectives stored In the syster^i? 



(b) test Items stored In the system? 



2* For future development of computer based testing programs, 
what are some of the ways the system could be manifested In 
the classroom? 

^change In the frequency of testing 

testing for Improvement of instruction 

teaching to specific objectives (for specific behaviors) 



jnore efficient-effective testing for grading purposes 

testing to describe the district 

testing to describe the children of the district 
_testlng to describe the instructional program of the 
district (behaviors which the program will elicit) 

other (please list): 



3. What have you actually Observed about teachers and/or classrooms 
that illustrates the impact of the present COMBAT system? 



As a district representative to MATPB: 

(a) How were you informed about CC^BAT operations? 



(b) Was this method of information transformation adequate? 

Did any special circumstances and/or constraints either 
help or hinder the operation of the COIBAT project? 

yes no 

(a) List those circumstances which helped the project management. 



(b) List those circumstances which hindered the project management « 



What were the most critical project decisions and 
how did the decisions affect the project? 

(a) project decision 

effects 

(b) project decision ^ 

effects 

(c) project decision _ 

effects 

Did you agree with the majority of the project decisions? 
yes no 

What were the consequences of these project decisions for you? 

positive ^ 

negative 



8. Were you involved in project dec5.sion-making? 
yes no 



If yes, what types of decisions did you participate in making? 



Was yovr involvement: by choice by assignment 

Would you have preferred: more involvement in decision-making 

less involvement in decision-making 

9. Rate the efficiency of project management on the basis of its 

expenditure of the following resources by putting a check mark in 
the appropriate box. 

DEGREE OF EFFICIENCY 



(a) 


consulting with MATPB 
representatives 


Very 
Low 

□ 


Low 

□ 


Average 

□ 


High 

□ 


Very 
High 

□ 


(b) 


use of teachers for 
system development 


□ 


□ 


□ 


n 


□ 


(c) 


use of computer facil- 
ities 


□ 


□ 


□ 


□ 


□ 


(d) 


use of project staff 


□ 


□ 


□ 


□ 


□ 


(e) 


use of budgeted 
financial resources 


□ 


□ 


□ 


□ 


□ 


(f) 


other 


□ 


□ 


□ 


□ 


□ 



10. The following list of activities were designed by the COMBAT staff 
to help develop test items and objectives and/or to stimulate use 
of the COMBAT system. Based on what you have OBSERVED, rate the 
effectiveness of each activity by checking the appropriate box. 

DEGREE OF EFFECTIVENESS 

Very Very 

Low Low Average High High 

(a) Summer, 1968-COMBAT 
teacher workshop for 

development of test I 11 | I I I | I I 

items and objectives J— J 1 — I l_J I — II — I 

(b) Fall, 1968-COMBAT 
initial experimental 

trials in Social Studies C] Q []] CH CD 

for grades 4-6. 

^3- 



10, Continued (Check the appropriate box) 



DEGREE OF EFFECTIVENESS 

Very Very 
Low Low Average High High 

(c) Fall & Winter, 1968-69 
teacher workshops on 
Measuring Educational 
Objectives offered 

through the Department I 11 11 I n I I 

of Continuing Education ' — ' ' — ' ' — ' ' — ' ' — ' 



(d) Fall.& Winter, 1969-70 
building and/or district 
level CCMBAT teacher 
orientation meetings 



□ □ □ □ □ 



(e) October, 1969~COMBAT 

teacher advisory team U LJ LJ LJ I I 

dinners 

11. What other strategies mi^t have been employed to Increase 
the use of the COMBAT system in the participating schools? 



12. The COMBAT staff used several additional means to distribute infor- 
mation about the COMBAT system and to build Interest. For each item, 
listed below, rate its effectiveness as a means of improving use of 
the system. Check the appropriate box. 

DEGREE OF EFFECTIVENESS 



ERIC 



(a) distribute COMBAT 
brochures 

(b) distribute telephone 
stickers advertising 
COMBAT 

(c) COMBAT slide- tape 
presentations 

(d) COMBAT article in 
OREGON EDUCATION 
December, 1969 

(e) COMBAT presentation 
at O.E.A. State 
Convention, 1969 



Very 








Very 


Low 


Low 


Average 


High 


High 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 




□ 


□ 



-4- 



12. 



Continued (Check the appropriate box) 



(£) distribution of 
matchbooks adver- 
tising COMBAT 



DEGREE OF EFFECTIVENESS 

Very Very 

Low Low Average High Hl^ 

□ □ n □ □ 



(g) distribution of 

COMBAT puzzles LJ d [U CD CU 

13, Generally, how do you rate the project tnanagemen t? 

Very Very 

Low Low Average . Higfi High 

Effectiveness ^ j~j j — j | — | j— j 

Efficiency □ □ □ □ □ 

List specific incidents, examples, problems, and/or decisions 
which contribute to your rating of the management. 



14. Generally, how do you rate the total project? 





Very 








Very 


Useful 


Low 


Low 


Average 


Hlgji 


High 


□ 


□ 


□ 


□ 


□ 


Important 


□ 


□ 


□ 


□ 


□ 


Significant 


□ 


□ 


□ 


□ 


□ 


Effective 


□ 




□ 


□ 


□ 


Feasible 


□ 


□ 


□ 


□ 


□ 



15. List one incident and/or example which illustrates your 
rating on each of the dimensions listed above. 

Useful 

Important^ 

Significan t 

Effectiv e 

? ^^^^^ -* 

Feasible 



ERIC 
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Appendix D 
Cover Letters for Questionnaires 



ERIC 



TEACHING RESEARCH 

A Division of the Oregon State System of Higher Education 
MONMOUTH, OREGON 97361 
Telephone (503)838-1220 



May 18, 1970 



Dear 



The COMBAT project, sponsored by the Metropolitan Area Testing 
Program Board, is currently terminating its third year of operation. 
The federal program under which it has been funded, ESEA, requires that 
it be subjected to an independent evaluation at this time. This evalu- 
ation is to provide the United States Office of Education with information 
which may be used to make decisions about funding other such programs 
throughout the United States* 

You were randomly identified, from among those persons who had used 
the COMBAT system, to assist us in this evaluative effort* We have pur- 
posely kept the sample size rather small, consequently, your responses to 
the instruments become extremely important. In effect, your response will 
serve as the opinion of eight teachers in the analysis process* 

Will you please complete the attached instrument today? When com- 
pleted , seal the instrument in its envelope, and return it to your principal* 

Needless to say, your cooperation in this important effort is greatly 
appreciated* 



I tfi ni«M*i«^i<ti t' In Mt IIdImi t'lhif iImim is i«>i4M*<*H *I iif Oh ('mh Si III lni%«»Ml\. I tti«« « Mfy* (iit|'«M«. mtUhUnt I lt« Nf, «li« #t fi<iili{ (m«i) 

t*i« MO I mHi lit t ililf •iiiiii NMtlliiiii Oil . 1.(1 I iilli, I I ut Oh Miitt < i« < Mi imt I .i littH .il hi<.t(tiit< . Piitll imi St it 1 i htv* hI ( i.iiiMMiti 

)t|iM,iMi*tl .lltil 'I 'Ml Ihih U(«M%tMh !>HOM»ll 



Yours truly. 




Frank G* Nelson, Director 
Combat Evaluation Team 



PGN/cj 



TEACHING RESEARCH 

A Divbion of the Oregon State System of Higher Education 
MONMOUTH, OREGON 97361 
Telephone (503)8384220 

June 11, 1970 



As you probably know, a requirement of all Title III projects is an 
independent or "third party" evaluation. The Teaching Research 
Evaluation Unit has contracted with the COMBAT project to fulfill 
this requirement. 

Two purposes are being pursued in our evaluation efforts* First, we 
are attempting to gather information that will be useful in making 
future operations of COMBAT and similar projects more effective. 
Second, we need to provide information to the U.S.O.E. which will 
satisfy their "accountability" requirements. Your position as a 
district representative to MATFB makes you uniquely qualified to 
provide us with information which can serve both purposes. 

We realize that you are very busy, but would appreciate a few minutes 
of your time to obtain answers to the attached questionnaire. Your 
response to each item is needed. If you feel that a specific question 
is not appropriate or applicable to your situation, we would like a 
brief explanation of why not. If you feel that any important aspect 
of the project evaluation, relating to your position as a district 
representative to MATPB, has been omitted please feel free to attach 
a written statement. These last two types of information will enable 
us to revise the evaluation design currently being employed. 

Because of the late date, we need to receive your response as soon 
as possible. Please try to complete the questionnaire and return it 
in the enclosed self-addressed staiiq>ed envelope within the next two 
days. 



erJc 



Thank you very much for your assistance. 



Sincere 




Robert R. Lange 
Asst. Director 
COMBAT Evaluation Team 

.i..«f r.iMm,'.if k.lm.Mtt.t.. Vi,i„|„ .tilt iff. !• isi, tti nr, m.h < olhw* . nrtK.Hi T.tl.nn.il hiMitiU, , IVhImmI Si H. I tif%M.fiv. ihi.sM... .»» i M.iMn.fnt, 



TEACHING RESEARCH 

A Division of the Oregon State System of Higher Education 
MONMOUTH, OREGON 97361 
Telephone (^05)838-1220 



May 18, 1970 



Dear 



The COMBAT project (Co^>uter Based Test Development), sponsored 
by the Metropolitan Area Testing Board (MATPD)^ Is teralnatlng its third 
year of operation. The Teaching Research Division of the OSSHE has been 
engaged to evaluate the progran according to the requlrenents of the 
Ihilted States Office of Education for projects funded under Title II, 
ESEA. The purpose of the evaluation Is to detemlne to vhat extent the < 
project set Its original objectives. The evaluation will also provide 
Information upon which the USOE say base future decisions regarding 
funding. We hope that the ultimate effect of a number of such evaluations 
will be to improve educational processes across the nation. 

During the paat three years COMBAT has developed a pool of Instruc- 
tional objectives and test Items for the use of teachers In Clackamas, 
Multnomah » and Washington counties. The objectives end test Items, 
prepared by teachers In these counties, have been s toted In a computer 
at the Portland Public Schools Data Processing Center. Teachers In 
the area have been able to telephone the center an4 x^uest both objec- 
tives and test Items for use In their own claasrooms^ At this time, 
the Item pool Includes objectives and test items for Social Studies, 
Language Arts, Science, and Mathematics. 

In the next few days, a ample of schools In th(^ COMBAT service 
area will be receiving short questlonnalrea dlstrlbutad and collected 
by the county lED offices. We need the aid of principals and teachers 
in your district in order to obtain the questionnaire data and appreciate 
any support you can give ua. If for some reason the schools in your 
district are unable to complete the distribution the questionnaires, 
we would appreciate your informing us aa soon as possible. 

Thank you for your cooperation. 



Sincerely yours. 




Frank Nelaon, Director 
COMBAT Evaluaticm Team 



FN:nr 



ERLC 
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TEACHING RESEARCH 

A Dtvisitn of the Oregon Stati System of Higher Education 
MONMOUTH, OREGOW 97361 
Telephone (505)858-1220 

May 18, 1970 



Dear 



The COMBaT project (Computer Based Tect Development) , sponsored 
by the Metropolitan Area Testing Board (MATPD), is terminating its third 
year of operation. The Teaching Research Division of the OSSHE has been 
engaged to evaluate the program according to the requirements of the 
United States Office of Education '^c projects funded under Title II, 
ESEA. The purpose of the evaluation is to ftetermine to what extent the 
project met its original objectives. The ..valuation will also provide 
information upon which the USOE may base future decisions regard^^ng 
funding. We hope that the t-ltimate effect of a number of such evaluations 
will be to improve educational processes across the nation. 

During the past three years COMBAT has developed a pool of in- 
structional objectives and test items for the use of teachers in 
Clackamas, Multnomah, and Washington counties. The objectives and test 
items, prepared by teachers in these counties, have been stored in a 
computer at the Portland Public Schools Data Processing Center. Teachers 
in the area have been able to telephone the center and request both 
objectives and test items for use in their own classrooms. At this time, 
the item pool includes objectives and test items for Social Studies, 
Language Arts, Science, and Mathematics. 

Your school has been selected by a random process, and we would 
appreciate your assistance very much. In a few days your courty lED 
will distribute packages of questionnaires to each selected school; and 
it would be most helpful to us if you would distribute the questionnaires 
to all teachers in grades four and above. We would also appreciate 
your urging all teache; to complete the questionnaires and return 
them upon the next day, so that the lED can collect them on schedule. 

Thank you for your cooperation. 

Sincerely yours, 



Frank Nelson, Director 
CO>fflAT Evaluation Team 



FN:bg 



. r- X"" S';"'" ?' "'B''" Ed"""'^''' i« tomptiieil of Oregon State Univmity. Univcriity jf Orcisun. inclii.tiiig The Mctticjl and f), ni,I School. 
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TEACHING RESEARCH 

A Dhisian of the Ongon Statt System of Higher Education 
MONMOUTH, OREGON 97361 
Tt J lone ()0})838-122O 

May 18, 1970 



Dear 

The COMBAT project (Computer Based Test Development), sponsored 
by the Metropolitan Area Testing Board (MATPD) , is terminating its third 
year of operation. The eaching Research Division of the OSSHE has been 
engaged to evaluate the program according to the requirements of the 
United States Office of Education for projects funded under Title II, 
ESEA. The purpose of the evaluation is to determine to what extent the 
project met its original objectives. The evaluation will also provide 
information upon which the USOE may base future decisions regarding 
funding. We hope that the ultimate effect of a number of such evalu- 
ations will be to improve educational processes across the nation. 

During the past three years COMBAT has developed a pool of in- 
structional objectives and test items for the use of teachers in 
Clackamas, Multnomah, and Washington counties. The objectives and test 
items, prepared by teachers in these counties, have been stored in a 
computer at the Portland Public Schools Data Processing Center. Teachers 
in the area have been able to telephone the center and request both 
objectives and test items for use in their own classrooms. At this time, 
the item pool includes objectives and test items for Social Studies, 
Language Arts, Science, and Mathematics. 

Your school has been selected by a random process, and we would 
appreciate your assistance very much. In a few days your county lED 
will distribute packages of questionnaires to each selected school; and 
it would be most helpful to us if you would distribute the questionnaires 
to all teachers in the department which will be indicated in your package. 
We would also appreciate your urging all teachers to complete the questionnaires 
and return them upon the next day, so that the lED can collect them on 
schedule. 

Thank you for your cooperation. 

Sincerely yours, 

Frank Nelson, Director 
COMBAT Evaluation Team 

h I\l L 3?* St«e S»Mm or Hisher Education i» comprlMd of Oregon State Univer»ity. University o{ Oregon. inctiKliiig The Medic»l amt Di-ntal ScIi.k.K. 

amim^Smm Ongon College of Education, SouUiem Oregon College. EaMern Ongon CoUege. Oregon Technical lB»titutc. Portland .Slate CollcKe. Diviiiiin of Contmuiiig 
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TEACHING RESEARCH 

A Division of the Oftgon State System of Higher Education 
MONMOUTH, OREGON 97361 
Telephooe (505)858*1220 



MEMORANDUM 



TO: Principals in the COMBAT Evaluation Sample 

FROM: COMBAT Evaluation Team, Teaching Research Division 

DATE: May 18, 1970 

Enclosed are the COMBAT questionnaires you have been expecting* Please 
distribute them to all teachers in grades four and above, and urge them 
to return the questionnaires as soon as possible* Each questionnaire 
should be returned to your office sealed by the teacher In the same envelope 
in which it comes. 

Would you please return the large envelope containing the completed 
questionnaires to the county lED office by May 27, so that we may gather 
the questionnaires from lED offices by Friday, May 29* 

We very much appreciate your efforts and the efforts of your teachers 
in behalf of the COMBAT evaluation. 



Sincerely yours. 




Frank 6* Nelson, Director 
COMBAT Evaluation Team 



FGN:cj 
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MONMOUTH, OREGON 97361 
May 18, 1970 



Dear Colleague: 

The C0>1BAT project, sponsored by the Metronolltan Area Testing 
Board (WPB), Is currently terminating its third year of operation. 
The word CO^IBAT stands for Computer-Based Test Development. During 
the past three years the project has developed a pool of instructional 
objectives and test items for the use of teachers in Clackamas, Wult- 
nomah and Washington counties. The objectives and test items, prepared 
by teachers in these counties, have been stored in a comnuter at the 
Portland Public Schools Data Processing Center. Teachers in the CO'^^BAT 
service area have been able to teJlephone the center and request both 
objectives and test items for use in their own classrooms. At this 
tine, the item pool includes objectives and test items for Social 
Studies. Language Arts, Science, and ^lathematics. 

The United States Office of Education requires that all projects 
funded under Title II, ESEA, such as CO'fBAT, be subjected to an inde- 
pendent evaluation. The purpose of this evaluation is to determine if 
the. project achieved its intended objectives and to provide the USOE 
with information needed for making decisions about funding future nro- 
grams. The ultimate effect of a number of such evaluation efforts can 
significantly improve education throughout the United States. 

Your school has been randomly identified, from among all the schools 
in the CO>tBAT service area, to assist us in this evaluation task. We 
have purposely kept the number of teachers involved as small as possible: 
consequently, your responses are extremely important. Tn effect, vour 
responses will be treated as the opinions of many other teachers. 

Would you please complete the attached instrument today? W\cn you 
have finished, seal the instrument in its envelono and return it to your 
prinripal. He will see that it is delivered to us. 

Your cooperation in this Important effort Is greatly appreciated. 



Sincerely, 




T^rnnk Vclson, director 



Wrhg 



